# NGINX Documentation > Learn how to deliver, manage, and protect your applications using F5 NGINX products. -------------------------------------------------------------------------------- title: "F5 NGINX Product Documentation" url: https://docs.nginx.com/ description: Learn how to deliver, manage, and protect your applications using F5 NGINX products. -------------------------------------------------------------------------------- # F5 NGINX Product Documentation Learn how to deliver, manage, and protect your applications using F5 NGINX products. ### F5 NGINX One ### [F5 NGINX One Console](/nginx-one-console/) Monitor your infrastructure, address security vulnerabilities, and assess the health of your NGINX fleet, all from a single console. ### [F5 NGINX Plus](/nginx/) The all-in-one load balancer, reverse proxy, web server, content cache, and API gateway. ### [F5 NGINX Instance Manager](/nginx-instance-manager/) Track and control NGINX Open Source and NGINX Plus instances. ### [F5 NGINX Ingress Controller](/nginx-ingress-controller/) Kubernetes traffic management with API gateway, identity, and observability features. ### [F5 NGINX Gateway Fabric](/nginx-gateway-fabric/) Next-generation Kubernetes connectivity using the Gateway API. ### [NGINX Open Source](https://nginx.org/en/docs/) The open source all-in-one load balancer, content cache, and web server. ### [NGINX Agent](/nginx-agent/) A daemon providing observability data and remote configuration for NGINX Open Source and NGINX Plus instances. ### [Subscription licensing & solutions](/solutions/) Stay compliant with your NGINX subscription licenses and see how you can use NGINX One to build secure, scalable, and high-performing applications and APIs. ### F5 WAF for NGINX ### [F5 WAF for NGINX](/waf/) Lightweight, high-performance, advanced protection against Layer 7 attacks on your apps and APIs. ### [F5 DoS for NGINX](/nginx-app-protect-dos/) Defend, adapt, and mitigate against Layer 7 denial-of-service attacks on your apps and APIs. ### F5 NGINXaaS ### [F5 NGINXaaS for AWS](/nginxaas/aws/overview/) Managed NGINX service for your AWS application stack. ### [F5 NGINXaaS for Google Cloud](/nginxaas/google/overview/) Managed NGINX service for your Google Cloud application stack. ### [F5 NGINXaaS for Azure](/nginxaas-azure/) Infrastructure-as-a-Service (IaaS) version of NGINX Plus for your Microsoft Azure application stack. ### Glossary ### [F5 NGINX Glossary](/glossary/) Glossary of common terms and definitions for all F5 NGINX products. -------------------------------------------------------------------------------- title: "F5 NGINX Glossary" url: https://docs.nginx.com/glossary/index.md -------------------------------------------------------------------------------- This glossary defines terms used in F5 NGINX. ## General terms | Term | Definition | | ---- | ---------- | | **Config Sync Group** / **Instance Group** | A group of NGINX systems (or instances) with identical configurations. They may also share the same certificates. However, the instances in a Config Sync Group could belong to different systems and even different clusters. Also known as an Instance Group in NGINX Instance Manager. For more information, see this explanation of [Important considerations](/nginx-one-console/nginx-configs/config-sync-groups/manage-config-sync-groups.md#important-considerations) | | **Control Plane** | The control plane is the part of a network architecture that manages and controls the flow or data or traffic (the Data Plane). It is responsible for system-level tasks such as routing and traffic management. | | **Data Plane** | The data plane is the part of a network architecture that carries user traffic. It handles tasks like forwarding data packets between devices and managing network communication. In the context of NGINX, the data plane is responsible for tasks such as load balancing, caching, and serving web content. | | **Instance** | An instance is an individual system with NGINX installed. You can group the instances of your choice in a Config Sync Group. When you add an instance to NGINX One Console, you need to use a data plane key. | | **Namespace** | In F5 Distributed Cloud, a namespace groups a tenant's configuration objects, similar to administrative domains. Every object in a namespace must have a unique name, and each namespace must be unique to its tenant. This setup ensures isolation, preventing cross-referencing of objects between namespaces. You'll see the namespace in the NGINX One Console URL as `/namespaces//`. To switch an instance between namespaces, you have to deregister an instance from an old namespace, and register it on the new namespace. | | **NGINX Agent** | A lightweight software component installed on NGINX instances to enable communication with the NGINX One console. NGINX Agent also enables communication with NGINX Instance Manager. | | **Staged Configurations** | Also known as **Staged Configs**. Allows you to save "work in progress." You can create it from scratch, an Instance, another Staged Config, or a Config Sync Group. It does _not_ have to be a working configuration until you publish it to an instance or a Config Sync Group. You can even manage your **Staged Configurations** through our [API](/nginx-one-console/api/api-reference-guide/#tag/StagedConfigs). | | **Tenant** | A tenant in F5 Distributed Cloud is an entity that owns a specific set of configuration and infrastructure. It is fundamental for isolation, meaning a tenant cannot access objects or infrastructure of other tenants. Tenants can be either individual or enterprise, with the latter allowing multiple users with role-based access control (RBAC). | ## Authentication and Authorization terms | Term | Definition | | ---- | ---------- | | **Access Token** | Defined in OAuth2, this (optional) short lifetime token provides access to specific user resources as defined in the scope values in the request to the authorization server (can be a JSON token as well). | | **ID Token** | Specific to OIDC, the primary use of the token in JWT format is to provide information about the authentication operation's outcome. | | **Identity Provider (IdP)** | A service that authenticates users and verifies their identity for client applications. | | **JSON Web Token (JWT)** | An open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. | | **Protected Resource** | A resource that is hosted by the resource server and requires an access token to be accessed. | | **Refresh Token** | Coming from OAuth2 specs, the token is usually long-lived and may be used to obtain new access tokens. | | **Relying Party (RP)** | A client service required to verify user identity. | ## Kubernetes and Ingress Controller terms {#k8s-ingress-controller} | Term | Definition | | ---- | ---------- | | **Ingress** | Refers to an *Ingress Resource*, a Kubernetes API object which allows access to [Services](https://kubernetes.io/docs/concepts/services-networking/service/) within a cluster. They are managed by an [Ingress Controller](/glossary/#ingress-controller). *Ingress* resources enable the following functionality:
* **Load balancing**, extended through the use of Services
* **Content-based routing**, using hosts and paths
* **TLS/SSL termination**, based on hostnames

For additional information, please read the official [Kubernetes Ingress Documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/). | | **Ingress Controller** | Ingress Controllers are applications within a Kubernetes cluster that enable [Ingress](/glossary/#ingress) resources to function. They are not automatically deployed with a Kubernetes cluster, and can vary in implementation based on intended use, such as load balancing algorithms for Ingress resources. [The design of NGINX Ingress Controller](/nic/overview/design.md) explains the technical details of NGINX Ingress Controller. | ## F5 WAF for NGINX This section defines terminology used when describing functionality of F5 WAF for NGINX. It assumes you are familiar with various layer 7 (L7) hypertext transfer protocol (HTTP) concepts such as: - Cookies - HTTP methods and status codes - HTTP headings, requests, responses, and parameters - Uniform Resource Identifier (URI) - Uniform Resource Location (URL) | Term | Definition | | ---- | ---------- | | Alarm | If selected, the F5 WAF for NGINX system records requests that trigger the violation in the remote log (depending on the settings of the logging profile). | | Attack signature | Textual patterns which can be applied to HTTP requests and/or responses by F5 WAF for NGINX to determine if traffic is malicious. For example, the string ` NGINX Logo

Server name: server_hostname

Server address: server_address

User Agent: client_browser

URI: server_url

Doc Root: document_root

Date: server_date

NGINX Frontend Load Balancer IP: remote_addr

Client IP: proxied_for_ip

NGINX Version: nginx_version

Auto Refresh
``` ## Configuring NGINX Plus for load balancing The steps in this section configure an NGINX Plus instance to load balance requests across the group of NGINX Open Source web servers you configured in the [previous section](#nginx-oss). If you are using these instructions to satisfy the prerequisites for one of our deployment guides, the Appendix in the guide specifies the names of the NGINX Plus instances used in it. Repeat these instructions on each instance. Alternatively, you can configure one instance and share the configuration with its peers in a cluster. See the [NGINX Plus Admin Guide](nginx/admin-guide/high-availability/configuration-sharing.md). 1. Open a connection to the NGINX Plus instance and change the directory to **/**etc/nginx/conf.d**: ```shell cd /etc/nginx/conf.d ``` 2. Rename **default.conf** to **default.conf.bak** so that NGINX Plus does not use it. ```shell mv default.conf default.conf.bak ``` 3. Create a new file called **lb.conf** with the following contents. **Note:** In the `upstream` blocks, include a [server](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#server) directive for each NGINX Open Source instance that serves the relevant application. ```nginx # in the 'http' context upstream app1 { server ; # 'server' directives for additional App 1 servers, if using zone app1 64k; } upstream app2 { server ; # 'server' directives for additional App 2 servers, if using zone app2 64k; } server { listen 80; status_zone backend; root /usr/share/nginx/html; location / { # directives for serving the site's HTML landing page } location /application1 { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; proxy_pass http://app1/; } location /application2 { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; proxy_pass http://app2/; } location /api { api write=on; # directives to control access, such as 'allow' and 'deny' } location = /dashboard.html { root /usr/share/nginx/html; } location = /status.html { # redirect requests that are made to pre-R14 dashboard return 301 /dashboard.html; } } ``` Directive documentation: [api](https://nginx.org/en/docs/http/ngx_http_api_module.html#api), [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen), [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location), [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass), [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header), [return](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return), [root](https://nginx.org/en/docs/http/ngx_http_core_module.html#root), [server](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#server) (upstream),[server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) (virtual), [server_name](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_name), [status_zone](https://nginx.org/en/docs/http/ngx_http_status_module.html#status_zone), [upstream](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream), [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) 4. Include the following directive in the top‑level ("main") context in **/etc/nginx/nginx.conf**, if it does not already appear there. ```nginx include conf.d/*.conf; ``` Directive documentation: [include](http://nginx.org/en/docs/ngx_core_module.html#include) ### Revision history - Version 2 (April 2019) – Generalized instructions for use with deployment guides - Version 1 (April 2018) – Initial version -------------------------------------------------------------------------------- title: "Single Sign-On with Amazon Cognito and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/cognito/index.md description: Enable OpenID Connect-based single-sign for applications proxied by NGINX Plus, using Amazon Cognito as the identity provider (IdP). -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With Amazon Cognito](nginx/deployment-guides/single-sign-on/cognito.md) for details. This guide explains how to enable single sign‑on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Amazon Cognito](https://aws.amazon.com/cognito/) as the identity provider (IdP), and NGINX Plus as the relying party. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites The instructions assume you have the following: - An [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/). - An NGINX Plus subscription and NGINX Plus R15 or later. For installation instructions, see the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (njs), required for handling the interaction between NGINX Plus and the IdP. After installing NGINX Plus, install the module with the command for your operating system. For Debian and Ubuntu: ```none sudo apt install nginx-plus-module-njs ``` For CentOS, RHEL, and Oracle Linux: ```shell sudo yum install nginx-plus-module-njs ``` - The following directive included in the top-level ("main") configuration context in **/etc/nginx/nginx.conf**, to load the NGINX JavaScript module: ```nginx load_module modules/ngx_http_js_module.so; ``` ## Configuring Amazon Cognito **Note:** The following procedure reflects the Cognito GUI at the time of publication, but the GUI is subject to change. Use this guide as a reference and adapt to the current Cognito GUI as necessary. Create a new application for NGINX Plus in the Cognito GUI: 1. Log in to your AWS account, open the AWS Management Console ([console.aws.amazon.com](https://console.aws.amazon.com)), and navigate to the Cognito dashboard (you can, for example, click **Cognito** in the **Security, Identity, & Compliance** section of the **Services** drop‑down menu). 2. On the Cognito dashboard, click **Manage User Pools** to open the **Your User Pools** window. Click the  Create a user pool  button or the highlighted phrase. 3. In the **Create a user pool** window that opens, type a value in the **Pool name** field (in this guide, it's **nginx-plus-pool**), then click the Review defaults button. 4. On the **Review** tab which opens, click Add app client... in the **App clients** field near the bottom. 5. On the **App clients** tab which opens, click Add an app client. 6. On the **Which app clients will have access to this user pool?** window which opens, enter a value (in this guide, **nginx-plus-app**) in the **App client name** field. Make sure the **Generate client secret** box is checked, then click the  Create app client  button. 7. On the confirmation page which opens, click **Return to pool details** to return to the **Review** tab. On that tab click the  Create pool  button at the bottom. (The screenshot in [Step 4](#cognito-review-tab) shows the button.) 8. On the details page which opens to confirm the new user pool was successfully created, make note of the value in the **Pool Id** field; you will add it to the NGINX Plus configuration in [Step 3 of _Configuring NGINX Plus_](#nginx-plus-variables). 'General settings' tab in Amazon Cognito GUI 9. Click **Users and groups** in the left navigation column. In the interface that opens, designate the users (or group of users, on the **Groups** tab) who will be able to use SSO for the app being proxied by NGINX Plus. For instructions, see the Cognito documentation about [creating users](https://docs.aws.amazon.com/cognito/latest/developerguide/how-to-create-user-accounts.html), [importing users](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-using-import-tool.html), or [adding a group](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-user-groups.html). 'Users and groups' tab in Amazon Cognito GUI 10. Click **App clients** in the left navigation bar. On the tab that opens, click the Show Details button in the box labeled with the app client name (in this guide, **nginx-plus-app**). 'App clients' tab in Amazon Cognito GUI 11. On the details page that opens, make note of the values in the **App client id** and **App client secret** fields. You will add them to the NGINX Plus configuration in [Step 3 of _Configuring NGINX Plus_](#nginx-plus-variables). 12. Click **App client settings** in the left navigation column. In the tab that opens, perform the following steps: 1. In the **Enabled Identity Providers** section, click the **Cognito User Pool** checkbox (the **Select all** box gets checked automatically). 2. In the **Callback URL(s)** field of the **Sign in and sign out URLs** section, type the URI of the NGINX Plus instance including the port number, and ending in **/\_codexch**. Here we’re using **https://my-nginx-plus.example.com:443/_codexch**. **Notes:** - For production, we strongly recommend that you use SSL/TLS (port 443). - The port number is mandatory even when you're using the default port for HTTP (80) or HTTPS (443). 3. In the **OAuth 2.0** section, click the **Authorization code grant** checkbox under **Allowed OAuth Flows** and the **email**, **openid**, and **profile** checkboxes under **Allowed OAuth Scopes**. 4. Click the  Save changes  button. 13. Click **Domain name** in the left navigation column. In the tab that opens, type a domain prefix in the **Domain prefix** field under **Amazon Cognito domain** (in this guide, **my-nginx-plus**). Click the  Save changes  button. ## Configuring NGINX Plus Configure NGINX Plus as the OpenID Connect relying party: 1. Create a clone of the [**nginx-openid-connect**](https://github.com/nginxinc/nginx-openid-connect) GitHub repository. ```shell git clone https://github.com/nginxinc/nginx-openid-connect ``` 2. Copy these files from the clone to **/etc/nginx/conf.d**: - **frontend.conf** - **openid_connect.js** - **openid_connect.server\_conf** 3. In your preferred text editor, open **/etc/nginx/conf.d/frontend.conf**. Change the second parameter of each of the following [set](http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#set) directives to the specified value. The `` variable is the full value in the **Domain prefix** field in [Step 13 of _Configuring Amazon Cognito_](#cognito-domain-name). In this guide it is **https://my-nginx-plus.auth.us-east-2.amazoncognito.com**. - `set $oidc_authz_endpoint` – `/oauth2/authorize` - `set $oidc_token_endpoint` – `/oauth2/token` - `set $oidc_client` – Value in the **App client id** field from [Step 11 of _Configuring Amazon Cognito_](#cognito-app-client-id-secret) (in this guide, `2or4cs8bjo1lkbq6143tqp6ist`) - `set $oidc_client_secret` – Value in the **App client secret** field from [Step 11 of _Configuring Amazon Cognito_](#cognito-app-client-id-secret) (in this guide, `1k63m3nrcnu...`) - `set $oidc_hmac_key` – A unique, long, and secure phrase 4. Configure the JWK file. The file's URL is **https://cognito-idp.**_region_**.amazonaws.com/**_User-Pool-ID_**/.well-known/jwks.json** where - _region_ is the same AWS region name as in the `` variable used in [Step 3](#nginx-plus-variables) - _User-Pool-ID_ is the value in the **Pool Id** field in [Step 8 of _Configuring Amazon Cognito_](#cognito-pool-id) In this guide, the URL is **https://cognito-idp.us-east-2.amazonaws.com/us-east-2_mLoGHJpOs/.well-known/jwks.json**. The method for configuring the JWK file depends on which version of NGINX Plus you are using: - In NGINX Plus R17 and later, NGINX Plus can read the JWK file directly. Change **/etc/nginx/conf.d/frontend.conf** as follows: 1. Comment out (or remove) the [auth_jwt_key_file](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) directive. 2. Uncomment the [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directive. (Its parameter, `/_jwks_uri`, refers to the value of the `$oidc_jwt_keyfile` variable, which you set in the next step.) 3. Change the second parameter of the `set $oidc_jwt_keyfile` directive to the URL of the JWK file (`https://cognito-idp.../.well-known/jwks.json`). - In NGINX Plus R16 and earlier, the JWK file must be on the local disk. (You can also use this method with NGINX Plus R17 and later if you wish.) 1. Copy the JSON contents from the JWK file (****) to a local file (for example, **/etc/nginx/my\_cognito\_jwk.json**). 2. In **/etc/nginx/conf.d/frontend.conf**, change the second parameter of the `set $oidc_jwt_keyfile` directive to the local file path. 5. At the time of publication, Cognito does not support the OpenID **offline_access** scope. Open **/etc/nginx/conf.d/openid\_connect.server\_conf** in a text editor and remove `+offline_access` from the list of scopes on line 10, so that it looks like this: ```nginx return 302 "$oidc_authz_endpoint?response_type=code&scope=openid+profile+email&client_id=$oidc_clientaws...; ``` 6. Confirm that the user named by the [user](http://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX Plus configuration (in **/etc/nginx/nginx.conf** by convention) has read permission on the JWK file. ## Testing In a browser, enter the address of your NGINX Plus instance and try to log in using the credentials of a user assigned to the application (see [Step 9 in _Configuring Amazon Cognito_](#cognito-users)). The NGINX logo that appears in the screenshot was added on Cognito's **UI customization** tab (not shown in this guide). ## Troubleshooting See the [**Troubleshooting**](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section at the **nginx-openid-connect** repository on GitHub. ### Revision History - Version 1 (March 2020) – Initial version (NGINX Plus Release 20) -------------------------------------------------------------------------------- title: "Single Sign-On With Auth0" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/auth0/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Auth0 as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Auth0](https://auth0.com/features/single-sign-on) as the Identity Provider (IdP), and NGINX Plus as the Relying Party, or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. In the latest NGINX Plus version, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - An [Auth0](https://auth0.com/) tenant with administrator privileges. - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Create a new Auth0 Application {#auth0-create} 1. Log in to your Auth0 Dashboard at [manage.auth0.com](https://manage.auth0.com/). 2. Select **Applications > Applications** from the sidebar menu. 3. On the **Applications** screen, select **Create Application**. 4. On the **Create application** screen: - Enter the **Name** for the application, for example, **NGINX Demo App**. - In **Application Type**, select **Regular Web Applications**. - Select **Create**. 5. On the **Settings** screen of your application: - Copy your **Client ID** and **Client Secret** displayed in the **Basic Information** section — you will need them later when configuring NGINX Plus. 6. On the **Application URIs** section of your application: - Add the NGINX Plus callback URI in the **Allowed Callback URLs** field, for example: `https://demo.example.com/oidc_callback`. - Add the post logout URL in the **Allowed Logout URLs** field, for example: `https://demo.example.com/post_logout/`. ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, Auth0 publishes the `.well-known/openid-configuration` document at the following address: `https://yourTenantId.us.auth0.com/.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https://yourTenantId.us.auth0.com/.well-known/openid-configuration | jq ``` Where: - the `yourTenantId` is your Auth0 [Tenant ID](https://auth0.com/docs/get-started/tenant-settings/find-your-tenant-name-or-tenant-id) - the `yourTenantId.us.auth0.com/` is your Auth0 server address - the `/.well-known/openid-configuration` is the default address for Auth0 for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://{yourTenantId}.us.auth0.com/", "authorization_endpoint": "https://{yourTenantId}.us.auth0.com/oauth/token", "token_endpoint": "https://{yourTenantId}.us.auth0.com/oauth/token", "jwks_uri": "https://{yourTenantId}.us.auth0.com/.well-known/jwks.json", "userinfo_endpoint": "https://{yourTenantId}.us.auth0.com/userinfo", "end_session_endpoint": "https://{yourTenantId}.us.auth0.com/oidc/logout", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for Auth0 is `https://yourTenantId.us.auth0.com/` (including the trailing slash). To verify the accuracy of the endpoints, refer to the [Auth0 official documentation](https://auth0.com/docs/get-started/applications/configure-applications-with-oidc-discovery). **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ## Set up NGINX Plus {#nginx-plus-setup} With Auth0 configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [Auth0 Configuration](#auth0-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the Auth0 provider named `auth0` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider auth0 { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - Your actual Auth0 **Client ID** obtained in [Auth0 Configuration](#auth0-create) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - Your **Client Secret** obtained in [Auth0 Configuration](#auth0-create) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - The **Issuer** URL obtained in [Auth0 Configuration](#auth0-create) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your Auth0 OIDC URL. For Auth0, a trailing slash is included, for example: `https://yourTenantId.us.auth0.com/`. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where Auth0 should redirect the user after a successful logout. This value **must also be specified** in **Allowed Logout URLs** on the Auth0 side. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user’s ID token as a *hint* to Auth0. This directive is **optional**, however, if it is omitted the Auth0 may display an extra confirmation page asking the user to approve the logout request. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch `/userinfo` from the Auth0 and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when Auth0's OpenID Connect discovery document advertises the `S256` code challenge method in the `code_challenge_methods_supported` field. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP metadata does not list `S256`. - The module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`), which remains the default for Auth0. - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider auth0 { issuer https://yourTenantId.us.auth0.com/; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with Auth0 OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `auth0` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#auth0-setup-oidc-provider): ```nginx # ... location / { auth_oidc auth0; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by Auth0: - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc auth0; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nAuth0 sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider auth0 { # Issuer from your Auth0 tenant's .well-known/openid-configuration issuer https://yourTenantId.us.auth0.com/; # Replace with your actual Client ID and Secret from Auth0 client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Enforce OIDC for root path with Auth0 auth_oidc auth0; # Forward OIDC claims to the upstream as headers if desired proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nAuth0 sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open `https://demo.example.com/` in a browser. You will be redirected to the Auth0 sign-in page. 2. Enter valid Auth0 credentials of a user who has access the application. Upon successful sign-in, Auth0 redirects you back to NGINX Plus, and you will see the proxied application content (for example, “Hello, Jane Doe!”). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; Auth0 ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to Auth0 for a fresh sign‑in, proving the session has been terminated. ## Legacy njs-based Auth0 Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based Auth0 Guide](nginx/deployment-guides/single-sign-on/oidc-njs/auth0.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added PKCE configuration (`pkce` directive) and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "Single Sign-On With Auth0 and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/auth0/index.md description: Learn how to enable single sign-on (SSO) with [Auth0](https://auth0.com/) for applications proxied by F5 NGINX Plus. -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With Auth0](nginx/deployment-guides/single-sign-on/auth0.md) for details. You can use F5 NGINX Plus with [Auth0](https://auth0.com/) and OpenID Connect to enable single sign-on (SSO) for your proxied applications. By following the steps in this guide, you will learn how to set up SSO using OpenID Connect as the authentication mechanism, with Auth0 as the identity provider (IdP), and NGINX Plus as the relying party. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites To complete the steps in this guide, you need the following: - An Auth0 tenant with administrator privileges. - [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) with a valid subscription. - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (`njs`) -- the `njs` module handles the interaction between NGINX Plus and Auth0. ## Install NGINX Plus and the njs Module {#install-nginx-plus-njs} 1. If you do not already have NGINX Plus installed, follow the steps in the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/) to do so. 2. Install the NGINX JavaScript module by following the steps in the [`njs` installation guide](https://nginx.org/en/docs/njs/install.html). 3. Add the following directive to the top-level ("main") configuration context in the NGINX Plus configuration (`/etc/nginx/nginx.conf`) to load the `njs` module: ```Nginx configuration file load_module modules/ngx_http_js_module.so; ``` ## Configure Auth0 {#config-auth0} Take the steps in this section to create a new application for NGINX Plus. **Note:** This section contains images that reflect the state of the Auth0 web interface at the time of publication. The actual Auth0 GUI may differ from the examples shown here. Use this guide as a reference and adapt the instructions to suit the current Auth0 GUI as necessary. ### Create a new Auth0 Application {#create-auth0-app} 1. Log in to your Auth0 Dashboard at [manage.auth0.com](https://manage.auth0.com/). 1. Select **Applications > Applications** from the sidebar menu. 1. On the **Applications** page, select the **Create Application** button. 1. In the **Create application** window, provide the information listed below and then select **Create**. - **Name**: A name for the application, for example "nginx-plus-app". - **Application Type**: **Regular Web Applications** ![image showing the Create application window in the Auth0 dashboard](/img/sso/auth0/sso-auth0-create-app.png) ### Set up the Web Application {#web-app-setup} In this section, you'll set up a web application that follows the Auth0 [Authorization Code Flow](https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow). 1. On the **Application** page in the [Auth0 dashboard](https://manage.auth0.com/), select your web application. 1. Select the **Settings** tab for your application. 1. Make note of the Client ID and Client Secret displayed in the **Basic Information** section. ![image showing the basic information section of the web application settings in the Auth0 dashboard](/img/sso/auth0/sso-auth0-app.png) 1. In the **Application URIs** section, provide the URI of the NGINX Plus instance in the **Allowed Callback URLs** field. - The URL must include the port number and end in **/_codexch**. In our example, we used the URL `http://nginx-plus-app:8010/_codexch`. - The port is always required, even if you use the default port for HTTP (`80`) or HTTPS (`443`). - The use of SSL/TLS (`443`) is strongly recommended for production environments. ![image showing the Application URIs settings in the Auth0 dashboard](/img/sso/auth0/sso-auth0-app-settings.png) 1. In the **Advanced Settings** section, select the **Endpoints** tab. 1. Make note of the **OpenID Configuration** URL. ![image showing the Advanced Application Settings section of the Auth0 dashboard](/img/sso/auth0/sso-auth0-app-advanced-settings.png) 1. Select **Save Changes**. ### Set up Authentication {#authn-setup} **Note:** For the purposes of this guide, we will add a new Auth0 user database and user account to use for testing. You can set up authentication using any of the available [Auth0 identity providers](https://auth0.com/docs/authenticate/identity-providers). To set up a new user database and add a user account to it, take the steps below. 1. Log in to the [Auth0 dashboard](https://manage.auth0.com/) and select **Authentication > Database** from the sidebar menu. 1. Select the **Create DB Connection** button. 1. Provide a **Name** for the database connection, then select **Create**. 1. On the **Database** page, select the **Applications** tab. Then, select the toggle button next to the [application you created earlier](#create-a-new-auth0-application). ![image showing the Applications settings for an OIDC Authentication database in the Auth0 dashboard](/img/sso/auth0/sso-auth0-db-app.png) 1. In the sidebar menu, select **User Management > Users**. 1. On the **Users** page, select the **Create User** button. 1. In the **Create user** window, provide the following information, then select **Create**. - **Email**: user's email - **Password**: a password for the user account - **Connection**: select your **database** from the list. ![image showing the Create User settings window in the Auth0 dashboard](/img/sso/auth0/sso-auth0-create-user.png) The user should receive an email to the email address provided. Once the user verifies their account by clicking on the link in the email, the account creation process is complete. ## Set up NGINX Plus {#nginx-plus-setup} Take the steps in this section to set up NGINX Plus as the OpenID Connect relying party. ### Configure NGINX OpenID Connect {#nginx-plus-oidc-config} 1. Clone the [nginx-openid-connect](https://github.com/nginxinc/nginx-openid-connect) GitHub repository, or download the repo files. ```bash git clone https://github.com/nginxinc/nginx-openid-connect.git ``` 1. Run the *configure.sh* script, which will update the NGINX configuration files with the values for your Auth0 application. For example: ```bash ./nginx-openid-connect/configure.sh \ --auth_jwt_key request \ --client_id Nhotzxx...IERmUi \ --client_secret 6ZHd0j_r...UtDZ5bkdu \ https://.us.auth0.com/.well-known/openid-configuration ``` 1. In the `frontend.conf` file, update the **my_backend** upstream with the address of the application that you want to add OIDC authorization to. For example: ```Nginx configuration file upstream my_backend { zone my_backend 64k; server my-backend-app.com:80; } ``` 1. In the *openid_connect.server_conf* file, add the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive to the `/_jwks_uri` and `/_token` locations to `Accept-Encoding "gzip"`, as shown below. ```Nginx configuration file ... location = /_jwks_uri { ... proxy_set_header Accept-Encoding "gzip" } ... location = /_token { ... proxy_set_header Accept-Encoding "gzip" } ... ``` 1. Copy the following files to the */etc/nginx/conf.d* directory on the host machine where NGINX Plus is installed: - `frontend.conf` - `openid_connect.js` - `openid_connect.server_conf` - `openid_connect_configuration.conf` 1. Reload the NGINX configuration: ```bash sudo nginx -s reload ``` ## Test the Setup 1. In a browser, enter the address of your NGINX Plus instance. You should be directed to the Auth0 login page, as shown in the example below. ![image showing an example Auth0 login screen that contains username and password fields](/img/sso/auth0/sso-auth0-login-test.png) 1. You should be able to log in using the credentials of the user account that you created in the Auth0 database. ## Troubleshooting Refer to the [Troubleshooting](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section in the `nginx-openid-connect` repository on GitHub. ## Revision History - Version 1 (May 2022) - Initial version -------------------------------------------------------------------------------- title: "Single Sign-On with Keycloak and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/keycloak/index.md description: Enable OpenID Connect-based single-sign for applications proxied by NGINX Plus, using Keycloak as the identity provider (IdP). -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With Keycloak](nginx/deployment-guides/single-sign-on/keycloak.md) for details. This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Keycloak](https://www.keycloak.org/) as the identity provider (IdP), and NGINX Plus as the relying party. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites The instructions assume you have the following: - A running Keycloak server. See the Keycloak documentation for [Getting Started](https://www.keycloak.org/guides#getting-started) and [Server](https://www.keycloak.org/guides#server) configuration instructions. - An NGINX Plus subscription and NGINX Plus R15 or later. For installation instructions, see the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (njs), required for handling the interaction between NGINX Plus and the IdP. After installing NGINX Plus, install the module with the command for your operating system. For Debian and Ubuntu: ```none sudo apt install nginx-plus-module-njs ``` For CentOS, RHEL, and Oracle Linux: ```shell sudo yum install nginx-plus-module-njs ``` - The following directive included in the top-level ("main") configuration context in **/etc/nginx/nginx.conf**, to load the NGINX JavaScript module: ```nginx load_module modules/ngx_http_js_module.so; ``` ## Configuring Keycloak **Note:** The following procedure reflects the Keycloak GUI at the time of publication, but the GUI is subject to change. Use this guide as a reference and adapt to the current Keycloak GUI as necessary. Create a Keycloak client for NGINX Plus in the Keycloak GUI: 1. Access the Keycloak Admin Console at **http://_keycloak-server-address_:8080/auth/admin/** and log in. 2. In the left navigation column, click **Clients**. On the **Clients** page that opens, click the **Create** button in the upper right corner. 3. On the **Add Client** page that opens, enter or select these values, then click the  Save  button. - **Client ID** – The name of the application for which you're enabling SSO (Keycloak refers to it as the “client”). Here we're using **NGINX-Plus**. - **Client Protocol** – **openid-connect**. 4. On the **NGINX Plus** page that opens, enter or select these values on the Settings tab: - **Access Type** – **confidential** - **Valid Redirect URIs** – The URI of the NGINX Plus instance, including the port number, and ending in **/\_codexch** (in this guide it is **https://my-nginx.example.com:443/_codexch**) **Notes:** - For production, we strongly recommend that you use SSL/TLS (port 443). - The port number is mandatory even when you're using the default port for HTTP (80) or HTTPS (443). 5. Click the Credentials tab and make a note of the value in the **Secret** field. You will copy it into the NGINX Plus configuration file in [Step 4 of _Configuring NGINX Plus_](#nginx-plus-variables). 6. Click the Roles tab, then click the **Add Role** button in the upper right corner of the page that opens. 7. On the **Add Role** page that opens, type a value in the **Role Name** field (here it is **nginx-keycloak-role**) and click the  Save  button. 8. In the left navigation column, click **Users**. On the **Users** page that opens, either click the name of an existing user, or click the **Add user** button in the upper right corner to create a new user. For complete instructions, see the [Keycloak documentation](https://www.keycloak.org/docs/latest/server_admin/index.html#user-management). 9. On the management page for the user (here, **user01**), click the Role Mappings tab. On the page that opens, select **NGINX-Plus** on the **Client Roles** drop‑down menu. Click **nginx-keycloak-role** in the **Available Roles** box, then click the **Add selected** button below the box. The role then appears in the **Assigned Roles** and **Effective Roles** boxes, as shown in the screenshot. ## Configuring NGINX Plus Configure NGINX Plus as the OpenID Connect relying party: 1. Create a clone of the [**nginx-openid-connect**](https://github.com/nginxinc/nginx-openid-connect) GitHub repository. ```shell git clone https://github.com/nginxinc/nginx-openid-connect ``` 2. Copy these files from the clone to **/etc/nginx/conf.d**: - **frontend.conf** - **openid\_connect.js** - **openid\_connect.server\_conf** - **openid\_connect\_configuration.conf** 3. Get the URLs for the authorization endpoint, token endpoint, and JSON Web Key (JWK) file from the Keycloak configuration. Run the following `curl` command in a terminal, piping the output to the indicated `python` command to output the entire configuration in an easily readable format. We've abridged the output to show only the relevant fields. ```shell $ curl https:///realms/master/.well-known/openid-configuration | python -m json.tool ... { "authorization_endpoint": "https:///realms/master/protocol/openid-connect/auth", ... "jwks_uri": "https:///realms/master/protocol/openid-connect/certs", ... "token_endpoint": "https:///realms/master/protocol/openid-connect/token", ... } ``` 4. Using your preferred text editor, open **/etc/nginx/conf.d/openid_connect_configuration.conf**. Change the "default" parameter value of each of the following [map](https://nginx.org/en/docs/http/ngx_http_map_module.html#map) directives to the specified value: - `map $host $oidc_authz_endpoint` – Value of `authorization_endpoint` from [Step 3](#nginx-plus-urls) (in this guide, `https:///realms/master/protocol/openid-connect/auth`) - `map $host $oidc_token_endpoint` – Value of `token_endpoint` from [Step 3](#nginx-plus-urls) (in this guide, `https:///realms/master/protocol/openid-connect/token`) - `map $host $oidc_client` – Value in the **Client ID** field from [Step 3 of _Configuring Keycloak_](#keycloak-client-id) (in this guide, `NGINX Plus`) - `map $host $oidc_client_secret` – Value in the **Secret** field from [Step 5 of _Configuring Keycloak_](#keycloak-secret) (in this guide, ``) - `map $host $oidc_hmac_key` – A unique, long, and secure phrase 5. Configure the JWK file. The procedure depends on which version of NGINX Plus you are using. - In NGINX Plus R17 and later, NGINX Plus can read the JWK file directly from the URL reported as `jwks_uri` in [Step 3](#nginx-plus-urls). Change **/etc/nginx/conf.d/frontend.conf** as follows: 1. Comment out (or remove) the [auth_jwt_key_file](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) directive. 2. Uncomment the [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directive. (Its parameter, `/_jwks_uri`, refers to the value of the `$oidc_jwt_keyfile` variable, which you set in the next step.) 3. Change the "default" parameter of the `map $host $oidc_jwt_keyfile` directive to the value reported in the `jwks_uri` field in [Step 3](#nginx-plus-urls) (in this guide, `https:///realms/master/protocol/openid-connect/certs`). - In NGINX Plus R16 and earlier, the JWK file must be on the local disk. (You can also use this method with NGINX Plus R17 and later if you wish.) 1. Copy the JSON contents from the JWK file named in the `jwks_uri` field in [Step 3](#nginx-plus-urls) (in this guide, `https:///realms/master/protocol/openid-connect/certs`) to a local file (for example, `/etc/nginx/my_keycloak_jwk.json`). 2. In **/etc/nginx/conf.d/openid_connect_configuration.conf**, change the "default" parameter of the `map $host $oidc_jwt_keyfile` directive to the local file path. 6. Confirm that the user named by the [user](http://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX Plus configuration (in **/etc/nginx/nginx.conf** by convention) has read permission on the JWK file. ## Testing In a browser, enter the address of your NGINX Plus instance and try to log in using the credentials of a user mapped to the role for NGINX Plus (see [Step 9 of _Configuring Keycloak_](#keycloak-users)). ## Troubleshooting See the [**Troubleshooting**](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section at the **nginx-openid-connect** repository on GitHub. ### Revision History - Version 2 (March 2020) – Updates to _Configuring NGINX Plus_ section - Version 1 (November 2019) – Initial version (NGINX Plus Release 19) -------------------------------------------------------------------------------- title: "Single Sign-On with Microsoft AD FS and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/active-directory-federation-services/index.md description: Enable OpenID Connect-based single-sign for applications proxied by NGINX Plus, using Microsoft AD FS as the identity provider (IdP). -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With Microsoft AD FS](nginx/deployment-guides/single-sign-on/active-directory-federation-services.md) for details. This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Microsoft Active Directory Federation Services](https://docs.microsoft.com/en-us/windows-server/identity/active-directory-federation-services) (AD FS) as the identity provider (IdP) and NGINX Plus as the relying party. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites The instructions assume you have the following: - A running deployment of AD FS, either on‑premises or in Azure. - An NGINX Plus subscription and NGINX Plus R15 or later. For installation instructions, see the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (njs), required for handling the interaction between NGINX Plus and the IdP. After installing NGINX Plus, install the module with the command for your operating system. For Debian and Ubuntu: ```none sudo apt install nginx-plus-module-njs ``` For CentOS, RHEL, and Oracle Linux: ```shell sudo yum install nginx-plus-module-njs ``` - The following directive included in the top-level ("main") configuration context in **/etc/nginx/nginx.conf**, to load the NGINX JavaScript module: ```nginx load_module modules/ngx_http_js_module.so; ``` ## Configuring AD FS Create an AD FS application for NGINX Plus: 1. Open the AD FS Management window. In the navigation column on the left, right‑click on the **Application Groups** folder and select **Add Application Group** from the drop‑down menu. The **Add Application Group Wizard** window opens. The left navigation column shows the steps you will complete to add an application group. 2. In the **Welcome** step, type the application group name in the **Name** field. Here we are using **ADFSSSO**. In the **Template** field, select **Server application** under **Standalone applications**. Click the  Next >  button. 3. In the **Server application** step: 1. Make a note of the value in the **Client Identifier** field. You will add it to the NGINX Plus configuration in [Step 4 of _Configuring NGINX Plus_](#nginx-plus-variables).
2. In the **Redirect URI** field, type the URI of the NGINX Plus instance including the port number, and ending in **/\_codexch**. Here we’re using **https://my-nginx.example.com:443/\_codexch**. Click the  Add  button. **Notes:** - For production, we strongly recommend that you use SSL/TLS (port 443). - The port number is mandatory even when you're using the default port for HTTP (80) or HTTPS (443). 3. Click the  Next >  button. 4. In the **Configure Application Credentials** step, click the **Generate a shared secret** checkbox. Make a note of the secret that AD FS generates (perhaps by clicking the **Copy to clipboard** button and pasting the clipboard content into a file). You will add the secret to the NGINX Plus configuration in [Step 4 of _Configuring NGINX Plus_](#nginx-plus-variables). Click the  Next >  button. 5. In the **Summary** step, verify that the information is correct, make any necessary corrections to previous steps, and click the  Next >  button. ## Configuring NGINX Plus Configure NGINX Plus as the OpenID Connect relying party: 1. Create a clone of the [**nginx-openid-connect**](https://github.com/nginxinc/nginx-openid-connect) GitHub repository. ```shell git clone https://github.com/nginxinc/nginx-openid-connect ``` 2. Copy these files from the clone to **/etc/nginx/conf.d**: - **frontend.conf** - **openid\_connect.js** - **openid\_connect.server\_conf** - **openid\_connect\_configuration.conf** 3. Get the URLs for the authorization endpoint, token endpoint, and JSON Web Key (JWK) file from the AD FS configuration. Run the following `curl` command in a terminal, piping the output to the indicated `python` command to output the entire configuration in an easily readable format. We've abridged the output to show only the relevant fields. ```shell $ curl https:///oidc/adfs/.well-known/openid-configuration | python -m json.tool { ... "authorization_endpoint": "https:///oidc/adfs/auth", ... "jwks_uri": "https:///oidc/adfs/certs", ... "token_endpoint": "https:///oidc/adfs/token", ... } ``` 4. In your preferred text editor, open **/etc/nginx/conf.d/frontend.conf**. Change the "default" parameter value of each of the following [map](https://nginx.org/en/docs/http/ngx_http_map_module.html#map) directives to the specified value: - `map $host $oidc_authz_endpoint` – Value of `authorization_endpoint` from [Step 3](#nginx-plus-urls) (in this guide, `https:///oidc/adfs/auth`) - `map $host $oidc_token_endpoint` – Value of `token_endpoint` from [Step 3](#nginx-plus-urls) (in this guide, `https:///oidc/adfs/token`) - `map $host $oidc_client` – Value in the **Client ID** field from [Step 3 of _Configuring AD FS_](#ad-fs-server-application) (in this guide, `3e23f0eb-9329-46ff-9d37-6ad24afdfaeb`) - `map $host $oidc_client_secret` – Value in the **Client secret** field from [Step 4 of _Configuring AD FS_](#ad-fs-configure-application-credentials) (in this guide, `NUeuULtSCjgXTGSkq3ZwEeCOiig4-rB2XiW_W`) - `map $host $oidc_hmac_key` – A unique, long, and secure phrase 5. Configure the JWK file. The procedure depends on which version of NGINX Plus you are using. - In NGINX Plus R17 and later, NGINX Plus can read the JWK file directly from the URL reported as `jwks_uri` in [Step 3](#nginx-plus-urls). Change **/etc/nginx/conf.d/frontend.conf** as follows: 1. Comment out (or remove) the [auth_jwt_key_file](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) directive. 2. Uncomment the [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directive. (Its parameter, `/_jwks_uri`, refers to the value of the `$oidc_jwt_keyfile` variable, which you set in the next step.) 3. Change the second parameter of the `set $oidc_jwt_keyfile` directive to the value reported in the `jwks_uri` field in [Step 3](#nginx-plus-urls) (in this guide, `https:///oidc/adfs/certs`). - In NGINX Plus R16 and earlier, the JWK file must be on the local disk. (You can also use this method with NGINX Plus R17 and later if you wish.) 1. Copy the JSON contents from the JWK file named in the `jwks_uri` field in [Step 3](#nginx-plus-urls) (in this guide, `https:///oidc/adfs/certs`) to a local file (for example, `/etc/nginx/my_adfs_jwk.json`). 2. In **/etc/nginx/conf.d/frontend.conf**, change the second parameter of the `set $oidc_jwt_keyfile` directive to the local file path. 6. Confirm that the user named by the [user](http://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX Plus configuration (in **/etc/nginx/nginx.conf** by convention) has read permission on the JWK file. ## Testing In a browser, enter the address of your NGINX Plus instance and try to log in using the credentials of a user who has access to the application. ## Troubleshooting See the [**Troubleshooting**](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section at the **nginx-openid-connect** repository on GitHub. ### Revision History - Version 2 (March 2020) – Updates to _Configuring NGINX Plus_ section - Version 1 (December 2019) – Initial version (NGINX Plus Release 20) -------------------------------------------------------------------------------- title: "Single Sign-On with Okta and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/okta/index.md description: Learn how to enable single sign-on (SSO) with Okta for applications proxied by F5 NGINX Plus. -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With Okta](nginx/deployment-guides/single-sign-on/okta.md) for details. You can use NGINX Plus with Okta and OpenID Connect to enable single sign-on (SSO) for your proxied applications. By following the steps in this guide, you will learn how to set up SSO using OpenID Connect as the authentication mechanism, with Okta as the identity provider (IdP), and NGINX Plus as the relying party. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites To complete the steps in this guide, you need the following: - An Okta administrator account. - [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) with a valid subscription. - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (`njs`) -- the `njs` module handles the interaction between NGINX Plus and Okta. - Install `jq` on the host machine where you installed NGINX Plus. ## Install NGINX Plus and the njs Module 1. If you do not already have NGINX Plus installed, follow the steps in the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/) to do so. 2. Install the NGINX JavaScript module by following the steps in the [`njs` installation guide](https://nginx.org/en/docs/njs/install.html). 3. Add the following directive to the top-level ("main") configuration context in the NGINX Plus configuration (`/etc/nginx/nginx.conf`) to load the `njs` module: ```Nginx configuration file load_module modules/ngx_http_js_module.so; ``` ## Configure Okta {#okta} Take the steps in this section to create a new application for NGINX Plus. **Note:** This section contains images that reflect the state of the Okta web interface at the time of publication. The actual Okta GUI may differ from the examples shown here. Use this guide as a reference and adapt the instructions to suit the current Okta GUI as necessary. This section describes the Okta Workforce Identity SSO product. You will need administrator access to your organization in Okta to complete this task. Your experience may differ somewhat if you're using the Okta Customer Identity product. ### Create a New Okta Web Application 1. Log in to Okta at [okta.com](https:///www.okta.com). 1. Select the **Admin** button next to your username to access the Admin console. 1. On your Admin dashboard, select **Applications** in the left-hand navigation menu. 1. On the **Applications** page, select the **Create App Integration** button. 1. In the **Create a new app integration** window, define the following values, then select **Next**: - **Sign-in method**: OIDC - OpenID Connect - **Application type**: Web Application ![image showing the Create a new app integration window in the Okta UI, with OIDC and Web Application options selected](/img/sso/okta/Okta-Create-New-Application-Integration.png) ### Set up the Web App Integration {#okta-integration} On the **New Web App Integration** page in the Okta web interface, fill in the following information, then select **Save**. | Field | Desciption | Example Value | |-------------|---------|----------| | **App integration name** | The name of the OpenID Connect relying party. Okta refers to this as the "application". | **NGINX-Plus** | | **Sign-in redirect URIs** | The URI of the NGINX Plus instance -- including the port number -- ending in **`/_codexch`**.
  • The port is always required, even if you use the default port for HTTP (`80`) or HTTPS (`443`).
  • The use of SSL/TLS (`443`) is strongly recommended for production environments.
| `https://my-nginx.example.com:443/_codexch` | | **Sign-out redirect URIs** | The URI to redirect users to after logging out.
This is an optional field with a default value of `http://localhost:8080`. | We removed the default value in our example. | | **Controlled access** | Controls who can access the application. | "Allow everyone in your organization to access"
**You should select the appropriate value for your use case.**| ![Okta Create OpenID Connect Integration](/img/sso/okta/Okta-Create-OpenID-Connect-Integration.png) ### Get the Okta App Client Credentials {#okta-client-id-secret} After you finish creating your application, the Okta Application page should display. You can find the Client Credentials for your Okta Application here. ![Image showing the application landing page in Okta, which contains the Client Credentials for the application.](/img/sso/okta/Okta-Client-Credentials.png) **Note:** If you need to find this information later, log in to your Okta admin account as [described above](#okta-login), select **Applications** in the left-hand menu, then select your application. Make note of the **Client ID** and **Client secret** values for your application. You will need these when you [configure NGINX Plus](#nginx-plus). ### Manage Access to your Okta Application {#okta-assign-applications} To change the users and groups that have access to your Okta Application: 1. Log in to Okta as an Admin as [described above](#okta-login). 1. Select **Applications** in the left-hand menu, then select your application. 1. Select the **Assignments** tab for the Application. Here, you can manage which users in your organization are granted access to this application. ## Set up NGINX Plus {#nginx-plus} Take the steps in this section to set up NGINX Plus as the OpenID Connect relying party. ### Configure NGINX OpenID Connect {#nginx-plus-oidc-config} 1. Clone the [nginx-openid-connect](https://github.com/nginxinc/nginx-openid-connect) GitHub repository, or download the repo files. ```shell git clone https://github.com/nginxinc/nginx-openid-connect.git ``` 1. Copy the following files to the `/etc/nginx/conf.d` directory on the host machine where NGINX Plus is installed: - `frontend.conf` - `openid_connect.js` - `openid_connect.server_conf` - `openid_connect_configuration.conf` 1. Get the URLs for the authorization endpoint, token endpoint, and JSON Web Key (JWK) file from the Okta configuration. Run the following `curl` command in a terminal. **Note:** We recommend piping the output to `jq` to output the entire configuration in an easily readable format. The output in the example below is abridged to show only the relevant fields. ```shell curl https://-admin.okta.com/.well-known/openid-configuration | jq ... { "authorization_endpoint": "https://.okta.com/oauth2/v1/authorize", ... "jwks_uri": "https://.okta.com/oauth2/v1/keys", ... "token_endpoint": "https://.okta.com/oauth2/v1/token", ... } ``` 1. Add the correct values for your IdP to the OpenID Connect configuration file (`/etc/nginx/conf.d/openid_connect_configuration.conf`). This file contains the primary configuration for one or more IdPs in `map{}` blocks. You should modify the `map…$oidc_` blocks as appropriate to match your IdP configuration. - Define the `$oidc_authz_endpoint`, `$oidc_token_endpoint`, and `$oidc_jwt_keyfile` values using the information returned in the previous step. - Change the URI defined in `map…$oidc_logout_redirect` to the URI of a resource (for example, your home page) that should be displayed after a client requests the `/logout` location. - Set a unique, long, and secure phrase for `$oidc_hmac_key` to ensure nonce values are unpredictable. ### Set up JSON Web Key Authorization {#nginx-plus-jwk-config} NGINX Plus can read the JWK file directly from the URL reported as `jwks_uri` in the output of the `curl` command you ran in the [previous section](#nginx-plus-oidc-config). **Note:** If you are using NGINX Plus R16 or earlier, refer to [Set up JWK Authorization using a local file](#nginx-plus-jwk-auth-local). Take the following steps to set up NGINX Plus to access the JWK file by using a URI. 1. In the `/etc/nginx/conf.d/frontend.conf` file, remove (or comment out) the [auth_jwt_key_file](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) directive. 1. Uncomment the [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directive. The parameter `/_jwks_uri` refers to the value of the `$oidc_jwt_keyfile` variable, which you already set in the OpenID Connect configuration file (`/etc/nginx/conf.d/openid_connect_configuration.conf`). #### Set up JWK Authorization using a Local File {#nginx-plus-jwk-auth-local} In NGINX Plus R16 and earlier, NGINX Plus cannot access the JWK file via the URI. Instead, the JWK file must be on the local disk. Take the steps below to set up JWK authorization using a local file: 1. Copy the JSON contents from the JWK file named in the `jwks_uri` field to a local file. For example, `/etc/nginx/my_okta_jwk.json` 1. In `/etc/nginx/conf.d/frontend.conf`, change the second parameter of the `set $oidc_jwt_keyfile` directive to the local file path of the JWK file. 1. Confirm that the user named by the [user](http://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX Plus configuration -- usually found in `/etc/nginx/nginx.conf` -- has read permission on the JWK file. ## Test Your Setup 1. In a browser, enter the address of your NGINX Plus instance. You should be directed to the okta login page, as shown in the example below. ![image](img/sso/okta/Okta-login-window.png) 1. Try to log in using the credentials of a user who is part of your organization. **Note:** If you restricted access to a group of users, be sure to select a user who has access to the application. ## Troubleshooting Refer to the [Troubleshooting](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section in the `nginx-openid-connect` repository on GitHub. ### Revision History - Version 3 (March 2022) – Full edit incorporating updates to _Configuring Okta_ and _Configuring NGINX Plus_ - Version 2 (March 2020) – Updates to _Configuring NGINX Plus_ section - Version 1 (April 2019) – Initial version (NGINX Plus Release 17) -------------------------------------------------------------------------------- title: "Single Sign-On with OneLogin and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/onelogin/index.md description: Learn how to enable single sign-on (SSO) with [OneLogin](https://www.onelogin.com/) for applications proxied by F5 NGINX Plus. -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With OneLogin](nginx/deployment-guides/single-sign-on/onelogin.md) for details. You can use NGINX Plus with [OneLogin](https://www.onelogin.com/) and the OpenID Connect protocol to enable single sign-on (SSO) for your proxied applications. By following the steps in this guide, you will learn how to set up SSO using OpenID Connect as the authentication mechanism, with OneLogin as the identity provider (IdP) and NGINX Plus as the relying party. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites To complete the steps in this guide, you need the following: - A OneLogin tenant with administrator privileges. - [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) with a valid subscription. - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (`njs`) -- the `njs` module handles the interaction between NGINX Plus and OneLogin identity provider (IdP). ## Install NGINX Plus and the njs Module {#install-nginx-plus-njs} 1. If you do not already have NGINX Plus installed, follow the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/) steps to do so. 2. Install the NGINX JavaScript module by following the steps in the [`njs` installation guide](https://nginx.org/en/docs/njs/install.html). 3. Add the following directive to the top-level ("main") configuration context in the NGINX Plus configuration (`/etc/nginx/nginx.conf`) to load the `njs` module: ```Nginx configuration file load_module modules/ngx_http_js_module.so; ``` ## Configure OneLogin {#config-onelogin} **Note:** The following procedure reflects the OneLogin GUI at the time of publication, but the GUI is subject to change. Use this guide as a reference and adapt to the current OneLogin GUI as necessary. Create a new application for NGINX Plus in the OneLogin GUI: 1. Log in to your OneLogin account at **https://**_domain_**.onelogin.com**, where _domain_ is the domain you chose when you created your account. 2. Click  Applications  in the title bar and then click the  Add App  button in the upper right corner of the window that opens. 3. On the **Find Applications** page that opens, type **OpenID Connect** in the search box. Click on the **OpenID Connect (OIDC)** row that appears. 4. On the **Add OpenId Connect (OIDC)** page that opens, change the value in the **Display Name** field to **NGINX Plus** and click the  Save  button. 5. When the save completes, a new set of choices appears in the left navigation bar. Click **Configuration**. In the **Redirect URI's** field, type the URI of the NGINX Plus instance including the port number, and ending in **/\_codexch** (in this guide it is **https://my-nginx.example.com:443/_codexch**). Then click the  Save  button. **Notes:** - For production, we strongly recommend that you use SSL/TLS (port 443). - The port number is mandatory even when you're using the default port for HTTP (80) or HTTPS (443). 6. When the save completes, click **SSO** in the left navigation bar. Click **Show client secret** below the **Client Secret** field. Record the values in the **Client ID** and **Client Secret** fields. You will add them to the NGINX Plus configuration in [Step 4 of _Configuring NGINX Plus_](#nginx-plus-variables). 7. Assign users to the application (in this guide, **NGINX Plus**) to enable them to access it for SSO. OneLogin recommends using [roles](https://onelogin.service-now.com/kb_view_customer.do?sysparm_article=KB0010606) for this purpose. You can access the **Roles** page under  Users  in the title bar. ## Set up NGINX Plus Take the steps in this section to set up NGINX Plus as the OpenID Connect Client. ### Configure NGINX OpenID Connect {#nginx-plus-oidc-config} 1. Clone the [nginx-openid-connect](https://github.com/nginxinc/nginx-openid-connect) GitHub repository, or download the repository files. ```shell git clone https://github.com/nginxinc/nginx-openid-connect.git ``` 1. Run the _configure.sh_ script to update the NGINX configuration files with the values for your OneLogin application. For example: ```bash ./nginx-openid-connect/configure.sh \ --auth_jwt_key request \ --client_id 168d5600-9224-... \ --client_secret c9210a67d09e85... \ https://.onelogin.com/oidc/2/.well-known/openid-configuration ``` 2. In the `frontend.conf` file, update the **my_backend** upstream with the address of the application that you want to add OIDC authorization to. For example: ```Nginx configuration file upstream my_backend { zone my_backend 64k; server my-backend-app.com:80; } ``` 3. In the _openid_connect.server_conf_ file, add the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive to the `/_jwks_uri` and `/_token` locations to `Accept-Encoding "gzip"`, as shown below. ```Nginx configuration file ... location = /_jwks_uri { ... proxy_set_header Accept-Encoding "gzip" } ... location = /_token { ... proxy_set_header Accept-Encoding "gzip" } ... ``` 4. Copy the following files to the _/etc/nginx/conf.d_ directory on the host machine where NGINX Plus is installed: - `frontend.conf` - `openid_connect.js` - `openid_connect.server_conf` - `openid_connect_configuration.conf` 5. Reload the NGINX configuration: ```bash sudo nginx -s reload ``` ## Test Your Setup In a browser, enter the address of your NGINX Plus instance and try to log in using the credentials of a user assigned to the application (see [Step 7 of _Configuring OneLogin_](#onelogin-roles)). ## Troubleshooting Refer to the [Troubleshooting](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section in the `nginx-openid-connect` repository on GitHub. ### Revision History - Version 3 (May 2022) - Updates OneLogin's OpenId Connect API endpoints from version 1 to version 2 - Version 2 (March 2020) – Updates to _Configuring NGINX Plus_ section - Version 1 (July 2019) – Initial version (NGINX Plus Release 18) -------------------------------------------------------------------------------- title: "Single Sign-On with Ping Identity and njs" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/oidc-njs/ping-identity/index.md description: Enable OpenID Connect-based single-sign for applications proxied by NGINX Plus, using Ping Identity as the identity provider (IdP). -------------------------------------------------------------------------------- **Note:** This guide applies to NGINX Plus [Release 15](nginx/releases.md#r15) and later, based on the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repo. Starting with NGINX Plus [Release 34](nginx/releases.md#r34), use the simpler solution with the [native OpenID connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). See [Single Sign-On With Ping Identity](nginx/deployment-guides/single-sign-on/ping-identity.md) for details. This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with Ping Identity as the identity provider (IdP) and NGINX Plus as the relying party. The instructions in this document apply to both Ping Identity's on‑premises and cloud products, PingFederate and PingOne for Enterprise. **Note:** You can find more information about the NGINX Plus OpenID Connect integration in the project's [GitHub repo](https://github.com/nginxinc/nginx-openid-connect#nginx-openid-connect). ## Prerequisites The instructions assume you have the following: - A running deployment of PingFederate or PingOne for Enterprise, and a Ping Identity account. For installation and configuration instructions, see the documentation for [PingFederate](https://docs.pingidentity.com/bundle/pingfederate-93/page/tau1564002955783.html) or [PingOne for Enterprise](https://docs.pingidentity.com/bundle/pingone/page/fjn1564020491958-1.html). - An NGINX Plus subscription and NGINX Plus R15 or later. For installation instructions, see the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - The [NGINX JavaScript module](https://nginx.org/en/docs/njs/index.html) (njs), required for handling the interaction between NGINX Plus and the IdP. After installing NGINX Plus, install the module with the command for your operating system. For Debian and Ubuntu: ```none sudo apt install nginx-plus-module-njs ``` For CentOS, RHEL, and Oracle Linux: ```shell sudo yum install nginx-plus-module-njs ``` - The following directive included in the top-level ("main") configuration context in **/etc/nginx/nginx.conf**, to load the NGINX JavaScript module: ```nginx load_module modules/ngx_http_js_module.so; ``` ## Configuring PingFederate or PingOne for Enterprise **Note:** This guide uses the GUI provided with PingOne for Enterprise. It reflects the GUI at the time of initial publication, but the GUI is subject to change. The PingFederate user interface might also differ. Use this guide as a reference and adapt as necessary for the UI you are using. Create a new application for NGINX Plus: 1. Log in to your Ping Identity account. The administrative dashboard opens automatically. In this guide, we show the PingOne for Enterprise dashboard, and for brevity refer simply to ”PingOne”. 2. Click  APPLICATIONS  in the title bar, and on the **My Applications** page that opens, click **OIDC** and then the **+ Add Application** button. 3. The **Add OIDC Application** window pops up. Click the ADVANCED CONFIGURATION box, and then the  Next  button. 4. In section 1 (PROVIDE DETAILS ABOUT YOUR APPLICATION), type a name in the **APPLICATION NAME** field and a short description in the **SHORT DESCRIPTION** field. Here, we're using **nginx-plus-application** and **NGINX Plus**. Choose a value from the **CATEGORY** drop‑down menu; here we’re using **Information Technology**. You can also add an icon if you wish. Click the  Next  button. 5. In section 2 (AUTHORIZATION SETTINGS), perform these steps: 1. Under **GRANTS**, click both **Authorization Code** and **Implicit**.
2. Under **CREDENTIALS**, click the **+ Add Secret** button. PingOne creates a client secret and opens the **CLIENT SECRETS** field to display it, as shown in the screenshot. To see the actual value of the secret, click the eye icon.
3. Click the  Next  button. 6. In section 3 (SSO FLOW AND AUTHENTICATION SETTINGS): 1. In the **START SSO URL** field, type the URL where users access your application. Here we’re using **https://example.com**. 2. In the **REDIRECT URIS** field, type the URI of the NGINX Plus instance including the port number, and ending in **/\_codexch**. Here we’re using **https://my-nginx-plus.example.com:443/\_codexch** (the full value is not visible in the screenshot). **Notes:** - For production, we strongly recommend that you use SSL/TLS (port 443). - The port number is mandatory even when you're using the default port for HTTP (80) or HTTPS (443). 7. In section 4 (DEFAULT USER PROFILE ATTRIBUTE CONTRACT), optionally add attributes to the required **sub** and **idpid** attributes, by clicking the **+ Add Attribute** button. We’re not adding any in this example. When finished, click the  Next  button. 8. In section 5 (CONNECT SCOPES), click the circled plus-sign on the **OpenID Profile (profile)** and **OpenID Profile Email (email)** scopes in the **LIST OF SCOPES** column. They are moved to the **CONNECTED SCOPES** column, as shown in the screenshot. Click the  Next  button. 9. In section 6 (ATTRIBUTE MAPPING), map attributes from your identity repository to the claims available to the application. The one attribute you must map is **sub**, and here we have selected the value **Email** from the drop‑down menu (the screenshot is abridged for brevity). 10. In section 7 (GROUP ACCESS), select the groups that will have access to the application, by clicking the circled plus-sign on the corresponding boxes in the **AVAILABLE GROUPS** column. The boxes move to the **ADDED GROUPS** column. As shown in the screenshot we have selected the two default groups, **Domain Administrators@directory** and **Users@directory**. Click the  Done  button. 11. You are returned to the **My Applications** window, which now includes a row for **nginx-plus-application**. Click the toggle switch at the right end of the row to the “on” position, as shown in the screenshot. Then click the “expand” icon at the end of the row, to display the application’s details. 12. On the page that opens, make note of the values in the following fields on the **Details** tab. You will add them to the NGINX Plus configuration in [Step 4 of _Configuring NGINX Plus_](#nginx-plus-variables). - **CLIENT ID** (in the screenshot, **28823604-83c5-4608-88da-c73fff9c607a**) - **CLIENT SECRETS** (in the screenshot, **7GMKILBofxb...**); click on the eye icon to view the actual value ## Configuring NGINX Plus Configure NGINX Plus as the OpenID Connect relying party: 1. Create a clone of the [**nginx-openid-connect**](https://github.com/nginxinc/nginx-openid-connect) GitHub repository. ```shell git clone https://github.com/nginxinc/nginx-openid-connect ``` 2. Copy these files from the clone to **/etc/nginx/conf.d**: - **frontend.conf** - **openid\_connect.js** - **openid\_connect.server\_conf** 3. Get the URLs for the authorization endpoint, token endpoint, and JSON Web Key (JWK) file from the Ping Identity configuration. Run the following `curl` command in a terminal, piping the output to the indicated `python` command to output the entire configuration in an easily readable format. We've abridged the output to show only the relevant fields. The `` variable is the value in the **CLIENT ID** field that you noted in [Step 12 of _Configuring PingFederate or PingOne for Enterprise_](#ping-client-id-secrets). **Note:** This `curl` command is appropriate for Ping One for Enterprise. For PingFederate, you might need to replace `sso.connect.pingidentity.com` with the IP address of your local PingFederate server. ```shell $ curl sso.connect.pingidentity.com//.well-known/openid-configuration | python -m json.tool ... { "authorization_endpoint": "https://sso.connect.pingidentity.com/sso/as/authorization.oauth2", ... "jwks_uri": "https://sso.connect.pingidentity.com/sso/as/jwks", ... "token_endpoint": "https://sso.connect.pingidentity.com/sso/as/token.oauth2", ... } ``` 4. In your preferred text editor, open **/etc/nginx/conf.d/frontend.conf**. Change the second parameter of each of the following [set](http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#set) directives to the specified value: - `set $oidc_authz_endpoint` – Value of `authorization_endpoint` from [Step 3](#nginx-plus-urls) (in this guide, `https://sso.connect.pingidentity.com/sso/as/authorization.oauth2`) - `set $oidc_token_endpoint` – Value of `token_endpoint` from [Step 3](#nginx-plus-urls) (in this guide, `https://sso.connect.pingidentity.com/sso/as/token.oauth2`) - `set $oidc_client` – Value in the **CLIENT ID** field in [Step 12 of _Configuring PingFederate or PingOne for Enterprise_](#ping-client-id-secrets) (in this guide, `28823604-83c5-4608-88da-c73fff9c607a`) - `set $oidc_client_secret` – Value in the **CLIENT SECRETS** field in [Step 12 of _Configuring PingFederate or PingOne for Enterprise_](#ping-client-id-secrets) (in this guide, `7GMKILBofxb...`) - `set $oidc_hmac_key` – A unique, long, and secure phrase 5. Configure the JWK file. The procedure depends on which version of NGINX Plus you are using. - In NGINX Plus R17 and later, NGINX Plus can read the JWK file directly from the URL reported as `jwks_uri` in [Step 3](#nginx-plus-urls). Change **/etc/nginx/conf.d/frontend.conf** as follows: 1. Comment out (or remove) the [auth_jwt_key_file](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) directive. 2. Uncomment the [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directive. (Its parameter, `/_jwks_uri`, refers to the value of the `$oidc_jwt_keyfile` variable, which you set in the next step.) 3. Change the second parameter of the `set $oidc_jwt_keyfile` directive to the value reported in the `jwks_uri` field in [Step 3](#nginx-plus-urls) (in this guide, `https://sso.connect.pingidentity.com/sso/as/jwks`). - In NGINX Plus R16 and earlier, the JWK file must be on the local disk. (You can also use this method with NGINX Plus R17 and later if you wish.) 1. Copy the JSON contents from the JWK file named in the `jwks_uri` field in [Step 3](#nginx-plus-urls) (in this guide, `https://sso.connect.pingidentity.com/sso/as/jwks`) to a local file (for example, `/etc/nginx/my_ping_identity_jwk.json`). 2. In **/etc/nginx/conf.d/frontend.conf**, change the second parameter of the `set $oidc_jwt_keyfile` directive to the local file path. 6. Confirm that the user named by the [user](http://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX Plus configuration (in **/etc/nginx/nginx.conf** by convention) has read permission on the JWK file. ## Testing In a browser, enter the address of your NGINX Plus instance and try to log in using the credentials of a user assigned to the application (see [Step 10 of _PingFederate or PingOne for Enterprise_](#ping-group-access)). ## Troubleshooting See the [**Troubleshooting**](https://github.com/nginxinc/nginx-openid-connect#troubleshooting) section at the **nginx-openid-connect** repository on GitHub. ### Revision History - Version 2 (March 2020) – Updates to _Configuring NGINX Plus_ section - Version 1 (January 2020) – Initial version (NGINX Plus Release 20) -------------------------------------------------------------------------------- title: "SPNEGO" url: https://docs.nginx.com/nginx/admin-guide/dynamic-modules/spnego/index.md description: This module adds [SPNEGO](https://tools.ietf.org/html/rfc4178) support to F5 NGINX Plus. Currently, only Kerberos authentication via GSSAPI is supported. -------------------------------------------------------------------------------- The SPNEGO Auth module adds [SPNEGO](https://tools.ietf.org/html/rfc4178) support to F5 NGINX Plus. Currently, only Kerberos authentication via GSSAPI is supported. ## Installation 1. Check the [Technical Specifications](/nginx/technical-specs.md#dynamic-modules) page to verify that the module is supported by your operating system. 2. Make sure that your operating system is configured to retrieve binary packages from the official NGINX Plus repository. See installation instructions for your operating system on the [Installing NGINX Plus](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md) page. 3. Install the SPNEGO Auth module package `nginx-plus-module-auth-spnego` from the official NGINX Plus repository. For Amazon Linux 2, CentOS, Oracle Linux, and RHEL: ```shell sudo yum update && \ sudo yum install nginx-plus-module-auth-spnego ``` For Amazon Linux 2023, AlmaLinux, Rocky Linux: ```shell sudo dnf update && \ sudo dnf install nginx-plus-module-auth-spnego ``` For Debian and Ubuntu:: ```shell sudo apt update && \ sudo apt install nginx-plus-module-auth-spnego ``` For SLES: ```shell sudo zypper refresh && \ sudo zypper install nginx-plus-module-auth-spnego ``` For Alpine: ```shell apk add nginx-plus-module-auth-spnego ``` For FreeBSD: ```shell sudo pkg update && \ sudo pkg install nginx-plus-module-auth-spnego ``` ## Configuration After installation you will need to enable and configure the module in F5 NGINX Plus configuration file `nginx.conf`. 1. Enable dynamic loading of the module with the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive specified in the top-level (“`main`”) context: ```nginx load_module modules/spnego-http-auth-nginx-module.so; http { # ... } ``` 2. Perform additional configuration as required by the [module](https://github.com/stnoonan/spnego-http-auth-nginx-module). 3. Test the NGINX Plus configuration. In a terminal, type-in the command: ```shell nginx -t ``` Expected output of the command: ```shell nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf is successful ``` 4. Reload the NGINX Plus configuration to enable the module: ```shell nginx -s reload ``` ## More info - [NGINX module for HTTP SPNEGO auth GitHub poject](https://github.com/stnoonan/spnego-http-auth-nginx-module) - [NGINX dynamic modules](dynamic-modules.md) - [NGINX Plus technical specifications](/nginx/technical-specs.md) - [Uninstalling a dynamic module](uninstall.md) -------------------------------------------------------------------------------- title: "Uninstalling a dynamic module" url: https://docs.nginx.com/nginx/admin-guide/dynamic-modules/uninstall/index.md description: This section describes how to uninstall an F5 NGINX Plus dynamic module. -------------------------------------------------------------------------------- You may need to uninstall a dynamic module in NGINX Plus in several scenarios: - The module is being replaced with a newer or alternative version. For example, the [OpenTracing](opentracing.md) module has been replaced by the [OpenTelemetry](opentelemetry.md) module. The [Cookie-Flag](cookie-flag.md) dynamic module has been superseded by natively supported [`proxy_cookie_flags`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cookie_flags) directive. - The module is no longer supported, for example, the [NGINX ModSecurity WAF](nginx-waf.md) module. - The operating system is not supported by the module. See the the [Dynamic Modules](/nginx/technical-specs.md#dynamic-modules) section of the [NGINX Plus Technical Specifications](/nginx/technical-specs.md#dynamic-modules). - The module it is no longer required, uninstalling it helps optimize resource usage and reduce maintenance overhead. - The module is causing runtime errors or configuration conflicts that affect stability or performance. - Security or compliance policies require removal, particularly in regulated environments where only approved or certified modules are allowed. ## Instructions To uninstall a dynamic module, run the appropriate command for your operating system in a terminal, replacing `` with the actual package name, for example, `nginx-plus-module-cookie-flag`. - For Amazon Linux 2, CentOS, Oracle Linux, and RHEL: ```shell sudo yum remove ``` - For Amazon Linux 2023, AlmaLinux, Rocky Linux: ```shell sudo dnf remove ``` - For Debian and Ubuntu: ```shell sudo apt remove ``` - For SLES: ```shell sudo zypper remove ``` - For FreeBSD: ```shell sudo pkg delete ``` ## Configuration After uninstalling the package, you will need to disable the module in the NGINX configuration file. 1. In a text editor, open the NGINX Plus configuration file: - `/etc/nginx/nginx.conf` for Linux - `/usr/local/etc/nginx/nginx.conf` for FreeBSD 2. Disable dynamic loading of the module by removing the corresponding [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. 3. Remove all directives related to the dynamic module. 4. Save the configuration. 5. Test the NGINX Plus configuration. In a terminal, type-in the command: ```shell nginx -t ``` Expected output of the command: ```shell nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf is successful ``` 5. Reload the NGINX Plus configuration: ```shell nginx -s reload ``` ## More info - [NGINX modules reference](https://nginx.org/en/docs/) - [NGINX dynamic modules](dynamic-modules.md) - [NGINX Plus technical specifications](/nginx/technical-specs.md) -------------------------------------------------------------------------------- title: "XSLT" url: https://docs.nginx.com/nginx/admin-guide/dynamic-modules/xslt/index.md description: Modify XML code in response bodies using one or more XSLT stylesheets, with the XSLT dynamic module, supported by NGINX, Inc. -------------------------------------------------------------------------------- With the XSLT dynamic module, you can modify XML code in response bodies using one or more XSLT stylesheets. ## Installation 1. Check the [Technical Specifications](/nginx/technical-specs.md#dynamic-modules) page to verify that the module is supported by your operating system. 2. Make sure that your operating system is configured to retrieve binary packages from the official NGINX Plus repository. See installation instructions for your operating system on the [Installing NGINX Plus](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md) page. 3. Install the XSLT module package `nginx-plus-module-xslt` from the official NGINX Plus repository. For Amazon Linux 2, CentOS, Oracle Linux, and RHEL: ```shell sudo yum update && \ sudo yum install nginx-plus-module-xslt ``` For Amazon Linux 2023, AlmaLinux, Rocky Linux: ```shell sudo dnf update && \ sudo dnf install nginx-plus-module-xslt ``` For Debian and Ubuntu: ```shell sudo apt update && \ sudo apt install nginx-plus-module-xslt ``` For SLES: ```shell sudo zypper refresh && \ sudo zypper install nginx-plus-module-xslt ``` For Alpine: ```shell apk add nginx-plus-module-xslt ``` For FreeBSD: ```shell sudo pkg update && \ sudo pkg install nginx-plus-module-xslt ``` ## Configuration After installation you will need to enable and configure the module in F5 NGINX Plus configuration file `nginx.conf`. 1. Enable dynamic loading of the module with the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive specified in the top-level (“`main`”) context: ```nginx load_module modules/ngx_http_xslt_module.so; http { # ... } ``` 2. Perform additional configuration as required by the [module](https://nginx.org/en/docs/http/ngx_http_xslt_module.html). 3. Test the NGINX Plus configuration. In a terminal, type-in the command: ```shell nginx -t ``` Expected output of the command: ```shell nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf is successful ``` 4. Reload the NGINX Plus configuration to enable the module: ```shell nginx -s reload ``` ## More info - [NGINX `ngx_http_xslt_module` module reference](https://nginx.org/en/docs/http/ngx_http_xslt_module.html) - [NGINX dynamic modules](dynamic-modules.md) - [NGINX Plus technical specifications](/nginx/technical-specs.md) - [Uninstalling a dynamic module](uninstall.md) -------------------------------------------------------------------------------- title: "Installing NGINX Plus LTS" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus-lts/index.md description: Install F5 NGINX Plus LTS with step-by-step instructions for the base package on all supported distributions. -------------------------------------------------------------------------------- Since May 13, 2026, NGINX Plus is published in two release types: Long-Term Support (LTS) and Continuous Release (CR). The NGINX Plus LTS release model is designed for mission-critical production environments. Each LTS release is supported for three years and receives security fixes and CVE mitigations without introducing new features. New features are delivered in CRs during the same annual LTS cycle. - **Cadence**: one LTS per year - **Patching model**: security/CVE fixes only, no feature changes - **Support window**: up to 3 years for each LTS release - **Concurrency**: up to 3 LTS versions supported at the same time - **Version format**: an LTS release has `0` as the second numeric component, for example: `PLS.37.0.0.1`. LTS updates increment the third component, for example: `PLS.37.0.1.1`. NGINX Plus CRs are published several times within an annual LTS cycle. Each CR contains the latest features and performance improvements. The CR cycle ends when a new LTS is released. ## Repository configuration options {#repo-options} By default, NGINX Plus repositories are configured to receive [Continuous Releases](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md). To use LTS, update your repository configuration to point to the LTS package URL, replacing the default URL. You can choose one of the options during installation: - **Pin to current LTS only**: receive only security updates for this LTS, no CRs, no upgrade to next LTS; supported up to three years. Follow the steps for your operating system in this guide. - **Pin to LTS track**: upgrade to the newest LTS when it is released annually, no CRs. Follow the steps for your operating system in this guide. - **Default**: receive Continuous Releases within the current LTS release, upgrade to each new LTS when it is released annually and then receive its CRs. See [Installing NGINX Plus](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md). ## Prerequisites {#prereq} Before you begin, make sure you have: - [MyF5 Customer Portal](https://account.f5.com/myf5) access, credentials are provided in the email from F5, Inc. - An active NGINX Plus subscription. Details can be verified on the [MyF5 Customer Portal](https://account.f5.com/myf5). - A [supported operating system and architecture](nginx/technical-specs.md). - Administrative privileges: `root` access or `sudo`, or see [Unprivileged installation](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#unpriv_install). - Internet access, or see [Offline installation](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#offline_install). ## Preparation steps for all operating systems {#common-steps} 1. Check if your operating system and architecture are supported. For a complete list of supported platforms and architectures, see the [Technical Specifications](nginx/technical-specs.md). 1. Back up your NGINX Plus configuration and log files if you have an older NGINX Plus package installed. For more information, see [Upgrading NGINX Plus](nginx/admin-guide/installing-nginx/upgrading-nginx-plus.md). 1. Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. 1. Create the **/etc/ssl/nginx** directory: ```shell sudo mkdir -p /etc/ssl/nginx ``` 1. Copy the downloaded **.crt** and **.key** files to the **/etc/ssl/nginx/** directory and make sure they are named **nginx-repo.crt** and **nginx-repo.key**: ```shell sudo cp .crt /etc/ssl/nginx/nginx-repo.crt sudo cp .key /etc/ssl/nginx/nginx-repo.key ``` 1. Follow the installation instructions for your operating system: [Amazon Linux 2023](#install_amazon2023), [Amazon Linux 2](#install_amazon2), [RHEL-based](#install_rhel), [Debian](#install_debian), [Ubuntu](#install_debian_ubuntu), [FreeBSD](#install_freebsd), [SLES](#install_suse), [Alpine](#install_alpine). ## Amazon Linux 2023 LTS packages {#install_amazon2023} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Install the **ca-certificates** dependency: ```shell sudo dnf update && \ sudo dnf install ca-certificates ``` 1. Add the NGINX Plus repository to your Amazon Linux 2023 instance. Download the [plus-amazonlinux2023.repo](https://cs.nginx.com/static/files/plus-amazonlinux2023.repo) file to **/etc/yum.repos.d**: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/plus-amazonlinux2023.repo ``` 1. **Modify your NGINX Plus repository configuration to pin to the desired LTS track**. To change your update channel, edit the `/etc/yum.repos.d/plus-amazonlinux2023.repo` file and update the `baseurl` to the [appropriate value](#repo-options) for your target version. - Pin to current LTS version: ```none baseurl=https://pkgs.nginx.com/plus/R37.0/amzn/2023/$basearch ``` - Pin to LTS track: ```none baseurl=https://pkgs.nginx.com/plus/LTS/amzn/2023/$basearch ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. ```shell sudo dnf install nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## Amazon Linux 2 LTS packages {#install_amazon2} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Install the **ca-certificates** dependency: ```shell sudo yum update && \ sudo yum install ca-certificates ``` 1. Add the NGINX Plus repository to your Amazon Linux 2 instance. Download the [nginx-plus-amazon2.repo](https://cs.nginx.com/static/files/nginx-plus-amazon2.repo) file to **/etc/yum.repos.d**: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/nginx-plus-amazon2.repo ``` 1. **Modify your NGINX Plus repository configuration to pin to the desired LTS track**. To change your update channel, edit the `/etc/yum.repos.d/nginx-plus-amazon2` file and update the `baseurl` to the [appropriate value](#repo-options) for your target version. - Pin to current LTS version: ```none baseurl=https://pkgs.nginx.com/plus/R37.0/amzn2/$releasever/$basearch ``` - Pin to LTS track: ```none baseurl=https://pkgs.nginx.com/plus/LTS/amzn2/$releasever/$basearch ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. ```shell sudo yum install nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## RHEL-based systems LTS packages {#install_rhel} Supported RHEL-based operating systems include Red Hat Enterprise Linux, Oracle Linux, AlmaLinux, and Rocky Linux for versions 8.1+, 9.7+, and 10+. 1. Make sure you meet the [prerequisites](#prereq) and have completed the [common steps for all operating systems](#common-steps). 1. Install the **ca-certificates** dependency: ```shell sudo dnf update && \ sudo dnf install ca-certificates ``` 1. Add the NGINX Plus repository by downloading **nginx-plus-.repo** file that matches your OS major version to **/etc/yum.repos.d**. - For **RHEL-based 8.1+**, download the [plus-8.repo](https://cs.nginx.com/static/files/plus-8.repo) file: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/plus-8.repo ``` - For **RHEL-based 9.7+**, download the [plus-9.repo](https://cs.nginx.com/static/files/plus-9.repo) file: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/plus-9.repo ``` - For **RHEL-based 10+**, download the [plus-10.repo](https://cs.nginx.com/static/files/plus-10.repo) file: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/plus-10.repo ``` 1. **Modify your NGINX Plus repository configuration to pin to the desired LTS track**. To change your update channel, edit the `/etc/yum.repos.d/plus-.repo` file and update the `baseurl` to the [appropriate value](#repo-options) for your target version.

For **RHEL-based 8.1+** - Pin to current LTS version: ```none baseurl=https://pkgs.nginx.com/plus/R37.0/centos/8/$basearch/ ``` - Pin to LTS track: ```none baseurl=https://pkgs.nginx.com/plus/LTS/centos/8/$basearch/ ``` For **RHEL-based 9.7+** - Pin to current LTS version: ```none baseurl=https://pkgs.nginx.com/plus/R37.0/centos/9/$basearch/ ``` - Pin to LTS track: ```none baseurl=https://pkgs.nginx.com/plus/LTS/centos/9/$basearch/ ``` For **RHEL-based 10+** - Pin to current LTS version: ```none baseurl=https://pkgs.nginx.com/plus/R37.0/centos/10/$basearch/ ``` - Pin to LTS track: ```none baseurl=https://pkgs.nginx.com/plus/LTS/centos/10/$basearch/ ``` - Save the changes and exit. - Update the repository information: ```shell sudo dnf update ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. ```shell sudo dnf install nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. To enable the NGINX service to start at boot, run the following command: ```shell sudo systemctl enable nginx.service ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## Debian LTS packages {#install_debian} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Install the prerequisites packages: ```shell sudo apt update && \ sudo apt install apt-transport-https \ lsb-release \ ca-certificates \ wget \ gnupg2 \ debian-archive-keyring ``` 1. Download and add NGINX signing key: ```shell wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key \ | gpg --dearmor \ | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null ``` 1. Add the NGINX Plus repository: ```shell printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://pkgs.nginx.com/plus/debian `lsb_release -cs` nginx-plus\n" \ | sudo tee /etc/apt/sources.list.d/nginx-plus.list ``` 1. Download the **nginx-plus** apt configuration to **/etc/apt/apt.conf.d**: ```shell sudo wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx ``` 1. **Modify your NGINX Plus repository configuration to pin to the desired LTS track**. To change your update channel, edit the `/etc/apt/sources.list.d/nginx-plus.list` file and update the URL to the [appropriate value](#repo-options) for your target version. - Pin to current LTS version: ```none https://pkgs.nginx.com/plus/R37.0/debian ``` - Pin to LTS track: ```none https://pkgs.nginx.com/plus/LTS/debian ``` 1. Update the repository information: ```shell sudo apt update ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. ```shell sudo apt install -y nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## Ubuntu LTS packages {#install_debian_ubuntu} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Install the prerequisites packages: ```shell sudo apt update && \ sudo apt install apt-transport-https \ lsb-release \ ca-certificates \ wget \ gnupg2 \ ubuntu-keyring ``` 1. Download and add NGINX signing key: ```shell wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key \ | gpg --dearmor \ | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null ``` 1. Add the NGINX Plus repository: ```shell printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://pkgs.nginx.com/plus/ubuntu `lsb_release -cs` nginx-plus\n" \ | sudo tee /etc/apt/sources.list.d/nginx-plus.list ``` 1. Download the **nginx-plus** apt configuration to **/etc/apt/apt.conf.d**: ```shell sudo wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx ``` 1. **Modify your NGINX Plus repository configuration to pin to the desired LTS track**. To change your update channel, edit the `/etc/apt/sources.list.d/nginx-plus.list` file and update the URL to the [appropriate value](#repo-options) for your target version. - Pin to current LTS version: ```none https://pkgs.nginx.com/plus/R37.0/ubuntu ``` - Pin to LTS track: ```none https://pkgs.nginx.com/plus/LTS/ubuntu ``` 1. Update the repository information: ```shell sudo apt update ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. ```shell sudo apt install -y nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## FreeBSD LTS packages {#install_freebsd} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Install the prerequisite **ca_root_nss** package: ```shell sudo pkg update && \ sudo pkg install ca_root_nss ``` 1. Copy the [nginx-plus.conf](https://cs.nginx.com/static/files/nginx-plus.conf) file to the **/etc/pkg/** directory: ```shell sudo fetch -o /etc/pkg/nginx-plus.conf http://cs.nginx.com/static/files/nginx-plus.conf ``` 1. Add the following lines to the **/usr/local/etc/pkg.conf** file: ```none PKG_ENV: { SSL_NO_VERIFY_PEER: "1", SSL_CLIENT_CERT_FILE: "/etc/ssl/nginx/nginx-repo.crt", SSL_CLIENT_KEY_FILE: "/etc/ssl/nginx/nginx-repo.key" } ``` 1. **Modify your NGINX Plus repository configuration to pin to the desired LTS track**. To change your update channel, edit the `/etc/pkg/nginx-plus.conf` file and update the `URL` to the [appropriate value](#repo-options) for your target version. - Pin to current LTS version: ```none URL: pkg+https://pkgs.nginx.com/plus/R37.0/freebsd/${ABI}/latest ``` - Pin to LTS track: ```none URL: pkg+https://pkgs.nginx.com/plus/LTS/freebsd/${ABI}/latest ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. Back up your NGINX Plus configuration and log files if you have an older NGINX Plus package installed. For more information, see [Upgrading NGINX Plus](#upgrade). ```shell sudo pkg install nginx-plus ``` 1. Copy the downloaded JWT file to the **/usr/local/etc/nginx** directory and make sure it is named **license.jwt**: ```shell sudo cp license.jwt /usr/local/etc/nginx ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/usr/local/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](https://docs.nginx.com/solutions/about-subscription-licenses/). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## SUSE Linux Enterprise LTS packages {#install_suse} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Create a file bundle of the certificate and key: ```shell cat /etc/ssl/nginx/nginx-repo.crt /etc/ssl/nginx/nginx-repo.key > /etc/ssl/nginx/nginx-repo-bundle.crt ``` 1. Install the required **ca-certificates** dependency: ```shell zypper refresh zypper install ca-certificates ``` 1. Add the **nginx-plus** repo. For **SLES 15**, pinned to current LTS version: ```shell zypper addrepo -G -t yum -c \ "https://pkgs.nginx.com/plus/R37.0/sles/15?ssl_clientcert=/etc/ssl/nginx/nginx-repo-bundle.crt&ssl_verify=peer" \ nginx-plus ``` For **SLES 15**, pinned to LTS track: ```shell zypper addrepo -G -t yum -c \ "https://pkgs.nginx.com/plus/LTS/sles/15?ssl_clientcert=/etc/ssl/nginx/nginx-repo-bundle.crt&ssl_verify=peer" \ nginx-plus ``` For **SLES 16**, pinned to current LTS version: ```shell zypper addrepo -G -t yum -c \ "https://pkgs.nginx.com/plus/R37.0/sles/16?ssl_clientcert=/etc/ssl/nginx/nginx-repo-bundle.crt&ssl_verify=peer" \ nginx-plus ``` For **SLES 16**, pinned to LTS track: ```shell zypper addrepo -G -t yum -c \ "https://pkgs.nginx.com/plus/LTS/sles/16?ssl_clientcert=/etc/ssl/nginx/nginx-repo-bundle.crt&ssl_verify=peer" \ nginx-plus ``` 1. Install the **nginx-plus** package. Any older NGINX Plus package is automatically replaced. ```shell zypper install nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## Alpine LTS packages {#install_alpine} 1. Make sure you have met all [prerequisites](#prereq) and completed the [common steps for all operating systems](#common-steps). 1. Upload **nginx-repo.key** to **/etc/apk/cert.key** and **nginx-repo.crt** to **/etc/apk/cert.pem**. Ensure these files contain only the specific key and certificate — Alpine Linux doesn't support mixing client certificates for multiple repositories. 1. Put the NGINX signing public key in the **/etc/apk/keys** directory: ```shell sudo wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub ``` 1. Add the NGINX repository to the **/etc/apk/repositories** file. - Pin to current LTS version: ```shell printf "https://pkgs.nginx.com/plus/R37.0/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" \ | sudo tee -a /etc/apk/repositories ``` - Pin to LTS track: ```shell printf "https://pkgs.nginx.com/plus/LTS/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" \ | sudo tee -a /etc/apk/repositories ``` 1. Remove all community-supported NGINX packages. Note that this will also remove all NGINX modules: ```shell sudo apk del -r nginx ``` 1. Install the NGINX Plus package: ```shell sudo apk add nginx-plus ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 1. Check the `nginx` version to verify that NGINX Plus LTS is installed correctly: ```shell nginx -v ``` The command output should indicate an LTS release: the second numeric component of the Plus release version should be `0`: ```none nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` 1. Make sure license reporting to F5 licensing endpoint is configured. By default, no configuration is required. However, it becomes necessary when NGINX Plus is installed in a disconnected environment, uses NGINX Instance Manager for usage reporting, or uses a custom path for the license file. Configuration can be done in the [`mgmt {}`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the NGINX Plus configuration file (`/etc/nginx/nginx.conf`). For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. If you are using [NGINX Instance Manager](/nim/) in your infrastructure, install and enable [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/). See [Install and Configure NGINX Agent](https://github.com/nginx/agent/blob/main/README.md) for details. ## Upgrade NGINX Plus {#upgrade} For general upgrade instructions, see [Upgrading NGNIX Plus](/nginx/admin-guide/installing-nginx/upgrading-nginx-plus.md). -------------------------------------------------------------------------------- title: "Upgrading NGINX Plus" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/upgrading-nginx-plus/index.md description: Upgrade F5 NGINX Plus with step-by-step instructions for the base package and dynamic modules on all supported Linux distributions. -------------------------------------------------------------------------------- This article explains how to upgrade existing NGINX Plus installation and dynamic modules. ## About {#upgrade} Keeping your NGINX Plus installation updated ensures it includes the latest features, security patches, and fixes. Critical bug patches and security updates are provided for the **two** most recent releases of NGINX Plus. Each NGINX Plus release reaches End of Software Development upon the next version's release, meaning no new features or routine bug fixes will be added to that version. ## Prerequisites Before upgrading, verify the following: 1. Your operating system is configured to retrieve binary packages from the official NGINX Plus repository: ensure that `nginx-plus.crt` and `nginx-plus.key` and repository file are set. See installation instructions for your operating system: [Amazon Linux 2023](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_amazon2023), [Amazon Linux 2](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_amazon2), [RHEL-based 8.1](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_rhel8), [RHEL-based 9](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_rhel), [RHEL-based 10](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_rhel10), [Debian](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_debian), [Ubuntu](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_debian_ubuntu), [FreeBSD](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_freebsd), [SLES](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_suse), [Alpine](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_alpine). 2. Your NGINX Plus subscription is active. You can verify your subscription on the [MyF5 Customer Portal](https://account.f5.com/myf5). 3. For NGINX Plus R33 and later, [license reporting](/solutions/about-subscription-licenses/getting-started.md) is configured. If upgrading from R32 or earlier, add the license file before upgrading and configure usage reporting. See [NGINX Plus R32 upgrade note](#nginx-plus-r32-and-earlier). ## Upgrade steps 1. Back up the configuration and log files. - For **Linux**: ```shell sudo cp -a /etc/nginx /etc/nginx-plus-backup && \ sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` - For **FreeBSD**: ```shell sudo cp -a /usr/local/etc/nginx /usr/local/etc/nginx-plus-backup && \ sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` 2. Upgrade to the newest NGINX Plus package. - For **RHEL-based**: ```shell sudo yum upgrade nginx-plus ``` - For **Debian** and **Ubuntu**: ```shell sudo apt update && \ sudo apt install nginx-plus ``` - For **FreeBSD**: ```shell sudo pkg upgrade nginx-plus ``` 3. Verify the upgrade: - Check the NGINX Plus version: ```shell nginx -v ``` The output of the command: ```shell nginx version: nginx/1.29.8 (nginx-plus-r37.0.0) ``` - Check the error log: ```shell tail /var/log/nginx/error.log ``` ## Upgrade notes ### NGINX Plus R24 and earlier Starting from [Release 24](nginx/releases.md#r24) (R24), NGINX Plus repositories have been separated into individual repositories based on operating system distribution and license subscription. Before upgrading from NGINX Plus R24 and earlier versions, you must first reconfigure your repositories to point to the correct location. To reconfigure your repository, follow the installation instructions above for your operating system: [Amazon Linux 2023](#install_amazon2023), [Amazon Linux 2](#install_amazon2), [RHEL-based 8.1](#install_rhel8), [RHEL-based 9](#install_rhel), [Debian or Ubuntu](#install_debian_ubuntu), [FreeBSD](#install_freebsd), [SLES](#install_suse). ### NGINX Plus R32 and earlier Starting from [NGINX Plus Release 33](nginx/releases.md#r33), a JWT license file is required for each NGINX Plus instance. For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). 1. Get the JWT file associated with your NGINX Plus subscription from the [MyF5 Customer Portal](https://account.f5.com/myf5): 1. Log in to [MyF5](https://my.f5.com/manage/s/). 1. Go to **My Products & Plans > Subscriptions** to see your active subscriptions. 1. Find your NGINX subscription, and select the **Subscription ID** for details. 1. Download the **JSON Web Token** file from the subscription page. 2. Create the **/etc/nginx/** directory for Linux or the **/usr/local/etc/nginx** directory for FreeBSD: - For **Linux**: ```shell sudo mkdir -p /etc/nginx ``` - For **FreeBSD**: ```shell sudo mkdir -p /usr/local/etc/nginx ``` **Note:** If you’re upgrading from NGINX Plus R32 or earlier to R33 or later and plan to use a custom path for the license file, note that the custom path isn’t recognized until after the upgrade. You must first create a placeholder file at `/etc/nginx/license.jwt` (or `/usr/local/etc/nginx/license.jwt` on FreeBSD). 1. **Before upgrading**: Create the placeholder file: ```bash touch /etc/nginx/license.jwt ``` 1. **After upgrading**: Update the [`license_token`](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive in the [`mgmt`](https://nginx.org/en/docs/ngx_mgmt_module.html) block of the configuration to point to your custom path: ```nginx mgmt { license_token ; } ``` 3. After downloading the JWT file, copy it to the **/etc/nginx/** directory for Linux, or to the **/usr/local/etc/nginx** directory for FreeBSD, and make sure it's named **license.jwt**: - For **Linux**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` - For **FreeBSD**: ```shell sudo cp .jwt /usr/local/etc/nginx/license.jwt ``` 4. Perform an upgrade. 5. After upgrade, it is possibly necessary to configure NGINX Plus usage reporting. By default, no configuration is required. However, if NGINX Plus is installed in an [offline environment](#offline_install) or if the JWT license file is located in a non-default directory, extra configuration is required. For offline environments, usage reporting should be configured for NGINX Instance Manager 2.18 or later. In the `nginx.conf` configuration file, specify the following directives: - the [`mgmt`](https://nginx.org/en/docs/ngx_mgmt_module.html#mgmt) context handles NGINX Plus licensing and usage reporting configuration, - the [`usage_report`](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive specifies the domain name or IP address of the NGINX Instance Manager, - the [`enforce_initial_report`](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive enables a 180-day grace period for sending the initial usage report. The initial usage report must be received by F5 licensing endpoint within this grace period. If the report is not received in time, traffic processing will be stopped: ```nginx mgmt { usage_report endpoint=NIM_FQDN; enforce_initial_report off; } ``` In NGINX Instance Manager, prepare and send the usage report to F5 licensing endpoint. For more information, see [Report usage to F5 in a disconnected environment](nim/licensing-and-reporting/report-usage-disconnected-deployment.md). If the JWT license file is located in a directory other than **/etc/nginx/** for Linux or **usr/local/etc/nginx/** for FreeBSD, you must specify its name and path in the [`license_token`](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive: ```nginx mgmt { license_token custom/file/path/license.jwt; } ``` ## Upgrade NGINX Plus modules {#upgrade_modules} The upgrade procedure depends on how the module was supplied and installed. - [NGINX‑authored](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#nginx-authored-dynamic-modules) and [NGINX‑certified community](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#nginx-certified-community-dynamic-modules) dynamic modules are updated automatically together with NGINX Plus. **Note:** For FreeBSD, each NGINX‑authored and NGINX‑certified module must be updated separately using FreeBSD package management tool. - [Community](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#community-dynamic-modules) dynamic modules must be recompiled against the corresponding NGINX Open Source version. See [Installing NGINX Community Modules](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#install_modules_oss). -------------------------------------------------------------------------------- title: "Active-Passive HA for NGINX Plus on AWS Using Elastic IP Addresses" url: https://docs.nginx.com/nginx/deployment-guides/amazon-web-services/high-availability-keepalived/index.md description: Create a highly available active-passive deployment of F5 NGINX Plus on AWS with a solution combining keepalived and the AWS Elastic IP address feature. -------------------------------------------------------------------------------- This guide explains how to create a high availability (HA) active‑passive deployment of F5 NGINX Plus in the [Amazon Web Services](https://aws.amazon.com/) (AWS) cloud. It combines the `keepalived`‑based solution for high availability (provided by NGINX for on‑premises HA deployments) with the AWS Elastic IP address feature. NGINX also provides a [solution for active‑active HA of NGINX Plus in AWS](high-availability-network-load-balancer.md), using AWS Network Load Balancer. ## Overview The [supported solution for HA deployment](nginx/admin-guide/high-availability/ha-keepalived.md) of NGINX Plus that uses `keepalived` is designed for on‑premises deployments. It is typically not viable in cloud environments, such as AWS, because of the networking restrictions they impose. One method for deploying NGINX Plus in a highly available manner on AWS is to use ELB in front of NGINX Plus instances. But, the method has several disadvantages: - It increases the cost of your deployment. - It limits the number of protocols NGINX Plus and your applications can support. In particular, ELB does not support UDP load balancing. - It does not provide a single static IP address for NGINX Plus instances, which is a crucial requirement for some applications. This guide explains how to create an active‑passive HA deployment of NGINX Plus on AWS that doesn’t require ELB and thus isn't subject to its disadvantages. It combines the `keepalived`‑based solution with AWS’s [Elastic IP address](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html) feature. Most importantly, this method addresses the requirement for a single IP address: as long as the primary NGINX Plus instance is operating correctly, it has the Elastic IP address. If the primary fails, the backup instance becomes the primary and reassociates the Elastic IP address with itself, as shown in the figure. When two NGINX Plus nodes hosted in AWS share an elastic IP address, the address switches to the backup automatically when the primary goes down, preserving high availability As an alternative to ELB, you can use Route 53 to distribute traffic among NGINX Plus instances, relying only on DNS load balancing. However, clients as well as intermediate DNS servers often cache DNS records as specified by the TTL value in the record, so there can be a delay in propagation of the updated records to the clients. This can lead to increased downtime of your applications as observed by clients. Such an update can happen when Route 53 detects the failure of an NGINX Plus instance and removes the corresponding record. In contrast, when you use the HA solution along with Route 53, the record usually doesn’t change because the IP address stays the same, and there is no TTL‑related problem. **Notes:** - We have successfully tested the instructions on Ubuntu 16.04 LTS (Xenial Xerus) and CentOS 7, with `keepalived` installed from the respective OS vendor repositories. - Except as noted, perform all steps on both the primary and backup instance. - The solution is not covered by your NGINX Plus support contract. - In addition to the [active‑active HA solution](high-availability-network-load-balancer.md) mentioned above, NGINX offers a solution based on [AWS Lambda](https://aws.amazon.com/lambda/) which does not require installation of any additional software on the NGINX Plus instances. The [F5 Professional Services](https://www.f5.com/services) team can deploy and configure the Lambda‑based solution for you and provide support. ## Step 1 – Launch Two NGINX Plus Instances The scripts in the HA solution use the AWS API to associate an Elastic IP address with an NGINX Plus instance, and so must have credentials for accessing the API. AWS uses _IAM roles_ to handle credentials, so you need to create a role and attach it to each NGINX Plus instance. Perform these steps (for step‑by‑step instructions, see the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)): 1. Create an IAM role and attach the following custom policy to it. The instance to which the policy applies can manipulate the Elastic IP address (adopt or release it) as well as perform the two indicated `Describe` actions. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:AssociateAddress", "ec2:DescribeInstances", "ec2:DescribeAddresses", "ec2:DisassociateAddress" ], "Resource": "*" } ] } ``` Alternatively, you can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to provide credentials to the HA scripts, as shown in [Step 5](#ha-aws_keepalived-configure). 2. Launch two instances and [install NGINX Plus](nginx/admin-guide/installing-nginx/installing-nginx-plus-amazon-web-services.md) on each. (As noted, we tested the instructions on Ubuntu 16.04 LTS and CentOS 7.) 3. Attach this IAM role to the instance. ## Step 2 – Allocate an Elastic IP address Allocate an Elastic IP address and remember its ID. For detailed instructions, see the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#using-instance-addressing-eips-allocating). ## Step 3 – Install `keepalived`, `wget`, and the AWS CLI 1. Install two packages from your OS vendor’s repository: **keepalived** and **wget**, which is used by the HA scripts. - On Ubuntu systems: ```shell sudo apt-get install keepalived wget ``` - On CentOS systems: ```shell sudo yum install keepalived wget ``` 2. Follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) to install the AWS CLI. ## Step 4 – Download the HA Scripts The NGINX Plus HA solution uses two scripts, which are invoked by `keepalived`: - **nginx-ha-check** – Determines the health of NGINX Plus. - **nginx-ha-notify** – Moves the Elastic IP address when a state transition happens, for example when the backup instance becomes the primary. 1. Create a directory for the scripts, if it doesn’t already exist. - On Ubuntu systems: ```shell sudo mkdir -p /usr/lib/keepalived ``` - On CentOS systems: ```shell sudo mkdir -p /usr/libexec/keepalived ``` 2. Download the scripts from our [GitHub repository](https://github.com/nginxinc/aws-ha-elastic-ip) into the created directory. ## Step 5 – Configure `keepalived` and the HA Scripts There are two configuration files for the HA solution: - **keepalived.conf** – The main configuration file for `keepalived`, slightly different for each NGINX Plus instance. - **nginx-ha-notify** – The script you downloaded in [Step 4](#ha-aws_ha-scripts), with several user‑defined variables. ### Creating keepalived.conf In the **/etc/keepalived** folder create a file named **keepalived.conf** with the following content. ```nginx vrrp_script chk_nginx_service { script "" interval 3 weight 50 } vrrp_instance VI_1 { interface eth0 priority virtual_router_id 51 advert_int 1 unicast_src_ip unicast_peer { } authentication { auth_type PASS auth_pass } track_script { chk_nginx_service } notify "" } ``` You must change values for the following configuration keywords. As you do so, also remove the angle brackets enclosing the placeholder value: - `script` in the `chk_nginx_service` block – The script that sends health checks to NGINX Plus. - On Ubuntu systems, **/usr/lib/keepalived/nginx-ha-check** - On CentOS systems, **/usr/libexec/keepalived/nginx-ha-check** - `priority` – The value that controls which instance becomes primary, with a higher value meaning a higher priority. Use `101` for the primary instance and `100` for the backup. - `unicast_src_ip` – This instance's IP address. - `unicast_peer` – The other instance's IP address. - `auth_pass` – The password string used for authentication between peers. - `notify` – The script that is invoked during a state transition. - On Ubuntu systems, **/usr/lib/keepalived/nginx-ha-notify** - On CentOS systems, **/usr/libexec/keepalived/nginx-ha-notify** ### Creating nginx-ha-notify Modify the user‑defined variables section of the **nginx-ha-notify** script, replacing each `` placeholder with the value specified in the list below: ```none export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_DEFAULT_REGION= HA_NODE_1= HA_NODE_2= ALLOCATION_ID= ``` - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` – The credentials for accessing the AWS API. Set them only when an IAM instance profile isn’t used. Otherwise, delete the corresponding two lines. - `AWS_DEFAULT_REGION` – The AWS region of your deployment. - `HA_NODE_1` and `HA_NODE_2` – The internal or private DNS names of the two NGINX Plus instances. - `ALLOCATION_ID` – The ID of the allocated Elastic IP address. ## Testing Run this command on both instances to start the `keepalived` daemon: ```shell sudo service keepalived start ``` The instance with the higher priority becomes the primary. As a result, the Elastic IP address becomes associated with the primary instance, as confirmed on the AWS Console. To check the instance state, run: ```shell cat /var/run/nginx-ha-keepalived.state ``` The command outputs `STATE=MASTER` on the primary instance and `STATE=BACKUP` otherwise. You can simulate the failure of the primary by stopping the `keepalived` daemon: ```shell sudo service keepalived stop ``` Check the state on the backup instance, confirming that it has transitioned to `MASTER`. Additionally, in the AWS Console the Elastic IP address is now associated with the new primary instance. ## Troubleshooting If the solution doesn’t work as expected, check the `keepalived` logs, which are written to **/var/log/syslog**. Also, you can manually run the commands that invoke the `awscli` utility in the **nginx-ha-notify** script to check that the utility is working properly. ## Caveats - In most of our tests it took 5 to 6 seconds for the Elastic IP address to be reassigned. - Elastic IP address reassignment is not free; see [Amazon EC2 Pricing](https://aws.amazon.com/ec2/pricing/). - Because the solution relies on the AWS APIs to reassociate the Elastic IP address, in some rare scenarios – such as flip‑flopping (the instances change state rapidly) or split‑brain (the instances lose connectivity with each other) – it is possible for the Elastic IP address not to end up associated with the primary. We were not able to reproduce these scenarios in our testing, however. If they occur, restart `keepalived` on both instances. ## Resources [Download the HA solution from the NGINX GitHub repository](https://github.com/nginxinc/aws-ha-elastic-ip). ### Revision History - Version 1 (May 2017) – Initial version (NGINX Plus Release 12) -------------------------------------------------------------------------------- title: "Configuring Active-Active High Availability and Additional Passive Nodes with keepalived" url: https://docs.nginx.com/nginx/admin-guide/high-availability/ha-keepalived-nodes/index.md description: Enable high availability of F5 NGINX Plus instances in on-premises deployments, with active-active pairs and multiple passive instances. -------------------------------------------------------------------------------- ## Introduction F5 NGINX Plus utilizes [keepalived](http://www.keepalived.org/) to provide high availability (HA) in a standard active‑passive fashion. This provides failover redundancy in the event of a problem on the primary NGINX Plus node. We can extend this functionality with additional nodes and changes to the `keepalived` configuration, providing additional redundancy and scalability options. This guide assumes that you have already configured NGINX Plus in an active‑passive implementation with [the NGINX HA solution](nginx/admin-guide/high-availability/ha-keepalived.md). **NOTE:** In a public cloud deployment we recommend using a Layer 4 or TCP load‑balancing service offered by the cloud provider to distribute traffic to NGINX Plus for active‑active functionality. ## Why Add a Passive Node? Many organizations have strict requirements on levels of redundancy and a two node active‑passive system may not meet these requirements. Adding a third node configured to take over in the event that both other nodes are down, provides further redundancy while keeping the configuration simple. This also allows for maintenance on a node without losing redundancy. ## Why Configure Active-Active HA? You can run NGINX Plus in an “active‑active” fashion, where two or more nodes handle traffic at the same time. This is achieved using multiple active IP addresses. Each IP address is hosted on a single NGINX instance, and the Keepalived configuration ensures that these IP addresses are spread across two or more active nodes. - When hosting multiple services, each service’s DNS name should resolve to one of the IP addresses. Share the IP addresses between the services. - Use round‑robin DNS to map a single DNS name to multiple IP addresses. - Use a L3 load‑balancing device such as a datacenter edge load balancer to distribute L3 traffic between the IP addresses. Active‑active may be used to increase the capacity of your load‑balanced cluster, but be aware that if a single node in an active‑active pair were to fail, the capacity would be reduced by half. You can use active‑Active as a form of safety, to provide sufficient resource to absorb unexpected spikes of traffic when both nodes are active, and you can use active‑active in larger clusters to provide more redundancy. Note that NGINX instances in a load‑balanced cluster do not share configuration or state. For best performance in an active‑active scenario, ensure that connections from the same client are routed to the same active IP address, and use session persistence methods such as [`sticky cookie`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_cookie) that do not rely on server‑side state. ## Configuring keepalived for an Additional Passive Node To configure an additional passive node for your existing NGINX Plus active‑passive HA pair, perform the following steps: 1. Install the **nginx-plus** and **nginx-ha-keepalived** packages on the new node. 2. Copy **/etc/keepalived/keepalived.conf** from the secondary node to the same location on the new node. 3. Edit **keepalived.conf** on the new node: - Lower the `priority` on any `vrrp_instance` blocks so that it is lower than the other nodes. - Change `unicast_src_ip` to match new node's host IP address. - Add the IP address of the secondary node to the `unicast_peer` section so that all other nodes are listed. Below is a sample **keepalived.conf** on an additional passive node with IP address 192.168.10.12. The IP addresses of the other two nodes are 192.168.10.10 and 192.168.10.11. The virtual IP address (VIP) is 192.168.10.100. ```yaml vrrp_script chk_nginx_service { script "/usr/lib/keepalived/nginx-ha-check" interval 3 weight 50 } vrrp_instance VI_1 { interface eth0 state BACKUP priority 99 virtual_router_id 51 advert_int 1 accept unicast_src_ip 192.168.10.12 unicast_peer { 192.168.10.10 192.168.10.11 } virtual_ipaddress { 192.168.10.100 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } ``` 4. Edit **keepalived.conf** on the other nodes, adding the IP address of the new passive node to the `unicast_peer` section so that all other nodes are listed: ```yaml unicast_peer { 192.168.10.11 192.168.10.12 } ``` 4. Restart `keepalived` on all nodes. 5. Test by stopping NGINX Plus on the first two nodes. All NGINX Plus nodes must have the identical configuration and SSL certificates. For information about synchronizing NGINX Plus configuration, see [Synchronizing NGINX Configuration in a Cluster](configuration-sharing.md). ## Configuring keepalived for Active-Active HA In order to direct traffic to both nodes at the same time, an additional VIP must be used. This new VIP will be active on the previously passive node, so that each node is active with its own VIP. To configure an existing NGINX Plus HA pair as active‑active, perform the following steps: 1. Edit **keepalived.conf** on the secondary node: - Copy the entire `vrrp_instance block VI_1` section and paste it below the existing block - In the copied `vrrp_instance` section: - Rename the new `vrrp_instance to VI_2` or another unique name - Change the `virtual_router_id` to `61` or another unique value - Change the `virtual_ipaddress` to an available IP address on the same subnet (in this example 192.168.10.101) - Change the `priority` value to `100` ```none vrrp_script chk_nginx_service { script "/usr/lib/keepalived/nginx-ha-check" interval 3 weight 50 } vrrp_instance VI_1 { interface eth0 state BACKUP priority 101 virtual_router_id 51 advert_int 1 accept unicast_src_ip 192.168.10.10 unicast_peer { 192.168.10.11 } virtual_ipaddress { 192.168.10.100 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } vrrp_instance VI_2 { interface eth0 state BACKUP priority 100 virtual_router_id 61 advert_int 1 accept unicast_src_ip 192.168.10.10 unicast_peer { 192.168.10.11 } virtual_ipaddress { 192.168.10.101 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } ``` 2. Edit **keepalived.conf** on the primary node: - Repeat the edits performed on the secondary node. - Set the `priority` within the new `vrrp_instance` to `99` or a value lower than on the secondary node. 3. Restart `keepalived` on all nodes. Configuration file and SSL certificate file synchronization is out of scope for this document but make sure all nodes have identical NGINX Plus configuration. ## Configuring NGINX Plus for Active-Active HA Now that the two NGINX Plus nodes are active with their own VIPs, NGINX Plus itself must be configured. There are two options for distributing traffic to the active nodes. Option 1 has all nodes active, with each node handling at least one application. Option 2 has all applications active on all nodes. **NOTE:** If the application being load balanced requires session persistence, we recommend that you use [sticky cookie](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_cookie), [sticky route](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_route) or [IP hash](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash) method, as they function correctly with multiple active nodes. [Sticky learn](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_learn) creates a session table in memory that is not shared between active nodes. ### Configuring NGINX Plus for Different Applications on Each Node In this configuration, each NGINX Plus node only processes requests to `server` blocks for which it has an active VIP. In the event of a failover, the active node is primary for additional VIPs and processes requests for the associated `server` blocks. Each `server` block includes the [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive to specify which VIP it is listening on. Building on the active‑active `keepalived` configuration from the [previous section](#conf_active), we're using the same two VIPs here. In this example application 1 is active on NGINX Plus node 1 and application 2 is active on NGINX node 2: ```nginx server { listen 192.168.10.100:80; location / { root /application1; } } server { listen 192.168.10.101:80; location / { root /application2; } } ``` ### Configuring NGINX Plus for All Applications on All Nodes In this configuration, NGINX Plus is able to process the traffic for any application on any VIP. In the event of a failure on a node, the VIP for that node moves to the node with the next highest priority. This way the DNS load balancing configuration does not need to change. Each NGINX Plus node listens for all requests. DNS load balancing is used to distribute requests to NGINX Plus nodes. Simple round-robin DNS is sufficient and can be configured according to the documentation for your DNS server. Ensure that there is an `A` for each VIP with the same fully qualified domain name (FQDN). Each time the name is resolved, the DNS server's response includes all VIPs, but in a different order. ```nginx server { listen *:80; location /app1 { root /application1; } location /app2 { root /application2; } } ``` ## Combining and Expanding Methods Both of the methods in [Configuring NGINX Plus for Active-Active HA](#nginx_for_active) can be combined for an active-active-passive configuration, or can even be extended to a configuration with any number of active nodes. ### Configuring All-Active HA on Three or More Nodes The following `keepalived` configuration is for an active-active-active configuration. Here the [steps for adding an active node](#conf_active) are just repeated for the third one. Notice that this node is active for one VIP, secondary for one VIP, and tertiary for one VIP. ```yaml vrrp_script chk_nginx_service { script "/usr/lib/keepalived/nginx-ha-check" interval 3 weight 50 } vrrp_instance VI_1 { interface eth0 state BACKUP priority 101 virtual_router_id 51 advert_int 1 accept unicast_src_ip 192.168.10.10 unicast_peer { 192.168.10.11 192.168.10.12 192.168.10.13 } virtual_ipaddress { 192.168.10.100 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } vrrp_instance VI_2 { interface eth0 state BACKUP priority 100 virtual_router_id 61 advert_int 1 accept unicast_src_ip 192.168.10.10 unicast_peer { 192.168.10.11 192.168.10.12 192.168.10.13 } virtual_ipaddress { 192.168.10.101 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } vrrp_instance VI_3 { interface eth0 state BACKUP priority 99 virtual_router_id 71 advert_int 1 accept unicast_src_ip 192.168.10.10 unicast_peer { 192.168.10.11 192.168.10.12 192.168.10.13 } virtual_ipaddress { 192.168.10.102 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } ``` ### Configuring Active-Active-Passive HA This example `keepalived` configuration is for the passive node in an active-active-passive configuration. It combines the steps in [Configuring keepalived for an Additional Passive Node](#conf_passive) and [Configuring keepalived for Active-Active HA](#conf_active). ```none vrrp_script chk_nginx_service { script "/usr/lib/keepalived/nginx-ha-check" interval 3 weight 50 } vrrp_instance VI_1 { interface eth0 state BACKUP priority 99 virtual_router_id 51 advert_int 1 accept unicast_src_ip 192.168.10.12 unicast_peer { 192.168.10.10 192.168.10.11 } virtual_ipaddress { 192.168.10.100 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } vrrp_instance VI_2 { interface eth0 state BACKUP priority 99 virtual_router_id 61 advert_int 1 accept unicast_src_ip 192.168.10.12 unicast_peer { 192.168.10.10 192.168.10.11 } virtual_ipaddress { 192.168.10.101 } track_script { chk_nginx_service } notify "/usr/lib/keepalived/nginx-ha-notify" } ``` -------------------------------------------------------------------------------- title: "Configuring Logging" url: https://docs.nginx.com/nginx/admin-guide/monitoring/logging/index.md description: Capture detailed information about errors and request processing in log files, either locally or via syslog. -------------------------------------------------------------------------------- Logging is essential for troubleshooting and understanding how traffic flows through your infrastructure. This article explains how to configure error and access logs in NGINX and NGINX Plus, customize access log formats, reduce noise with conditional logging, and forward logs to syslog for centralized collection. ## Set up the error log {#error_log} NGINX writes an error log that records issues of different severity levels. The [`error_log`](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive sets up the log location and severity level. The log location can be a particular file, `stderr`, or `syslog`. ### Default path By default, the error log is located at **logs/error.log**, but the absolute path depends on the operating system and installation method. You can find the path to error log by running `nginx -V` command and extracting the value of `--error-log-path=`: ```shell nginx -V 2>&1|sed -n 's/.*--error-log-path=\([^ ]*\).*/\1/p' ``` The output depends on the operating system. - For RHEL-based, Debian, Ubuntu: ```shell /var/log/nginx/error.log ``` - For FreeBSD: ```shell /usr/local/var/log/nginx/error.log ``` - For docker images, it is sent to container logs (stderr) via symlinks `/var/log/nginx/error.log -> /dev/stderr`. ### Severity level The error severity level follows the `syslog` classification system. The log includes messages from all severity levels above the specified level. The following example changes the minimal severity level of error messages to log from `error` to `warn`: ```nginx error_log logs/error.log warn; ``` With this setting, messages above the `warn` level are logged. That includes `warn`, `error`, `crit`, `alert`, and `emerg` levels. For debug logging, see [Debugging NGINX](debugging.md). ### Error log formats The `error_log` directive writes error messages in the standard NGINX text error-log format. In NGINX Plus, error log also supports structured JSON output via a [`json`](https://nginx.org/en/docs/ngx_core_module.html#error_log_json) parameter: ```nginx error_log /var/log/nginx/error.json error json; ``` Example of JSON-formatted error entry: ```json { "level": "error", "timestamp": "2026-05-04T10:30:15.042+00:00", "pid": 12345, "tid": 12345, "cnum": 3, "msg": "connect() failed", "client": "192.168.1.10", "server": "example.com", "request": "GET /api HTTP/1.1", "upstream": "http://127.0.0.1:8080/api", "errno": 111, "errtext": "Connection refused" } ``` ### Error log inheritance The default setting of the error log works globally. To override it, place the [`error_log`](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive in the `main` (top-level) configuration context. Settings in the `main` context are always inherited by other configuration levels (`http`, `server`, `location`). The `error_log` directive can also be specified on the [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http), [`stream`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream), `server` and [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) levels. Settings at lower levels override the settings inherited from the higher levels. Each error message is written only once to the error log closest to the level where the error has occurred. However, if several `error_log` directives are specified on the same level, the message is written to all specified logs. ## Set up the access log {#access_log} NGINX records client requests in the access log right after the request is processed. The [`access_log`](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directive specifies the location of the log and its format. By default, the access log is located at **logs/access.log**. The format of logged messages is the predefined **combined** format. To change the format of logged messages, use the [log_format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) directive. The log format is defined using variables. The following example extends the predefined **combined** format by specifying the `gzip ratio` keyword. This enables gzip compression of the response in a virtual server. ```nginx http { log_format compression '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" "$gzip_ratio"'; server { gzip on; access_log /spool/logs/nginx-access.log compression; ... } } ``` Another example of the log format enables tracking different time values between NGINX and an upstream server. This may help with diagnosing a slowdown of your website. You can use the following variables to log the indicated time values: - [`$upstream_connect_time`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_connect_time) – The time spent on establishing a connection with an upstream server - [`$upstream_header_time`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_header_time) – The time between establishing a connection and receiving the first byte of the response header from the upstream server - [`$upstream_response_time`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_response_time) – The time between establishing a connection and receiving the last byte of the response body from the upstream server - [`$request_time`](https://nginx.org/en/docs/http/ngx_http_log_module.html#var_request_time) – The total time spent processing a request All time values are measured in seconds with millisecond resolution. ```nginx http { log_format upstream_time '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"' 'rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"'; server { access_log /spool/logs/nginx-access.log upstream_time; ... } } ``` When reading the resulting time values, keep the following in mind: - When a request is processed through several servers, the variable contains several values separated by commas - When there is an internal redirect from one upstream group to another, the values are separated by semicolons - When a request is unable to reach an upstream server or a full header cannot be received, the variable contains `0` (zero) - In case of internal error while connecting to an upstream or when a reply is taken from the cache, the variable contains `-` (hyphen) The log can be optimized by enabling the `buffer` and the `cache`. With the [buffer](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) parameter enabled, messages will be stored in the buffer first. When the buffer is full (or in some other [cases](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log)), the messages will be written to the log. To enable caching of log file descriptors, use the [open_log_file_cache](https://nginx.org/en/docs/http/ngx_http_log_module.html#open_log_file_cache) directive. Similar to the `error_log` directive, the [access_log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directive defined on a particular configuration level overrides the settings from the previous levels. When processing of a request is completed, the message is written to the log that is configured on the current level, or inherited from the previous levels. If one level has multiple access log definitions, the message is written to all of them. ## Enabling conditional logging {#conditional} Conditional logging allows excluding trivial or unimportant log entries from the access log. In NGINX, conditional logging is enabled by the `if` parameter to the [access_log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directive. This example excludes requests with HTTP status codes `2xx` (Success) and `3xx` (Redirection): ```nginx map $status $loggable { ~^[23] 0; default 1; } access_log /path/to/access.log combined if=$loggable; ``` ## Usecase: sampling TLS parameters {#tls_sample} Many clients use TLS versions older than TLS 1.3. Though many ciphers are declared insecure, older implementations still use them; ECC certificates offer greater performance than RSA, but not all clients can accept ECC. Many TLS attacks rely on a “man in the middle” who intercepts the cipher negotiation handshake and forces the client and server to select a less secure cipher. Therefore, it’s important to configure F5 NGINX Plus to not support weak or legacy ciphers, but doing so may exclude legacy clients. You can evaluate the SSL data obtained from the client and determine what proportion of clients get excluded if support for older SSL protocols and ciphers is removed. The following configuration example logs the SSL protocol, cipher, and `User-Agent` header of any connected TLS client, assuming that each client selects the most recent protocol and most secure ciphers it supports. In this example, each client is identified by its unique combination of IP address and User-Agent. 1. Define the custom log format `sslparams` that includes the version of the SSL protocol ([`$ssl_protocol`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_protocol)), ciphers used in the connection ([`$ssl_cipher`](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_cipher)), the client IP address ([`$remote_addr`](http://nginx.org/ru/docs/http/ngx_http_core_module.html#var_remote_addr)), and the value of standard `User Agent` HTTP request field (`$http_user_agent`): ```nginx log_format sslparams '$ssl_protocol $ssl_cipher ' '$remote_addr "$http_user_agent"'; ``` 2. Define a key-value storage that will keep the IP address of the client and its User Agent, for example, `clients`: ```nginx keyval_zone zone=clients:80m timeout=3600s; ``` 3. Create a variable, for example, `$seen` for each unique combination of `$remote_addr` and `User-Agent` header: ```nginx keyval $remote_addr:$http_user_agent $seen zone=clients; server { listen 443 ssl; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers HIGH:!aNULL:!MD5; if ($seen = "") { set $seen 1; set $logme 1; } access_log /tmp/sslparams.log sslparams if=$logme; # ... } ``` 4. View the log file generated with this configuration: ```none TLSv1.2 AES128-SHA 1.1.1.1 "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 2.2.2.2 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 3.3.3.3 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:58.0) Gecko/20100101 Firefox/58.0" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 4.4.4.4 "Mozilla/5.0 (Android 4.4.2; Tablet; rv:65.0) Gecko/65.0 Firefox/65.0" TLSv1 AES128-SHA 5.5.5.5 "Mozilla/5.0 (Android 4.4.2; Tablet; rv:65.0) Gecko/65.0 Firefox/65.0" TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 6.6.6.6 "Mozilla/5.0 (Linux; U; Android 5.0.2; en-US; XT1068 Build/LXB22.46-28) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.108 UCBrowser/12.10.2.1164 Mobile Safari/537.36" ``` 5. Process the log file to determine the spread of data: ```shell cat /tmp/sslparams.log | cut -d ' ' -f 2,2 | sort | uniq -c | sort -rn | perl -ane 'printf "%30s %s\n", $F[1], "="x$F[0];' ``` In this output, low‑volume, less secure ciphers are identified: ```shell ECDHE-RSA-AES128-GCM-SHA256 ========================= ECDHE-RSA-AES256-GCM-SHA384 ======== AES128-SHA ==== ECDHE-RSA-CHACHA20-POLY1305 == ECDHE-RSA-AES256-SHA384 == ``` Then you can check the logs to determine which clients are using these ciphers and then make a decision about removing these ciphers from the NGINX Plus configuration. For more information about sampling requests with NGINX conditional logging see the [blog post](https://www.nginx.com/blog/sampling-requests-with-nginx-conditional-logging/#var_request_id). ## Logging to syslog {#syslog} The `syslog` utility is a standard for computer message logging and allows collecting log messages from different devices on a single syslog server. In NGINX, logging to syslog is configured with the `syslog:` prefix in [error_log](https://nginx.org/en/docs/ngx_core_module.html#error_log) and [access_log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directives. Syslog messages can be sent to a `server=` which can be a domain name, an IP address, or a UNIX-domain socket path. A domain name or IP address can be specified with a port to override the default port, `514`. A UNIX-domain socket path can be specified after the `unix:` prefix: ```nginx error_log syslog:server=unix:/var/log/nginx.sock debug; access_log syslog:server=[2001:db8::1]:1234,facility=local7,tag=nginx,severity=info; ``` In the example, NGINX error log messages are written to a UNIX domain socket at the `debug` logging level, and the access log is written to a syslog server with an IPv6 address and port `1234`. The `facility=` parameter specifies the type of program that is logging the message. The default value is `local7`. Other possible values are: `auth`, `authpriv`, `daemon`, `cron`, `ftp`, `lpr`, `kern`, `mail`, `news`, `syslog`, `user`, `uucp`, `local0 ... local7`. The `tag=` parameter applies a custom tag to syslog messages (`nginx` in our example). The `severity=` parameter sets the severity level of syslog messages for access log. Possible values in order of increasing severity are: `debug`, `info`, `notice`, `warn`, `error` (default), `crit`, `alert`, and `emerg`. Messages are logged at the specified level and all more severe levels. In our example, the severity level `error` also enables `crit`, `alert`, and `emerg` levels to be logged. ## Live activity monitoring {#monitoring} NGINX Plus provides a real-time live activity monitoring interface that shows key load and performance metrics of your [HTTP](nginx/admin-guide/load-balancer/http-load-balancer.md) and [TCP](nginx/admin-guide/load-balancer/tcp-udp-load-balancer.md) upstream servers. See the [Live Activity Monitoring](live-activity-monitoring.md) article for more information. ## Compatibility note {#compat} - Support for multiple `error_log` directives at the same configuration level was introduced in NGINX Open Source [1.5.2](https://nginx.org/en/CHANGES). - Support for JSON-formatted error logs was introduced in NGINX Plus [PLS.37.0.0.1 LTS](/nginx/releases.md#r37.0). -------------------------------------------------------------------------------- title: "Control NGINX Processes at Runtime" url: https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/index.md description: Understand the NGINX processes that handle traffic, and how to control them at runtime. -------------------------------------------------------------------------------- This section describes the processes that NGINX starts at run time and how to control them. ## Master and worker processes {#processes} NGINX has one master process and one or more worker processes. If [caching](nginx/admin-guide/content-cache/content-caching.md) is enabled, the cache loader and cache manager processes also run at startup. The main purpose of the master process is to read and evaluate configuration files, as well as maintain the worker processes. The worker processes do the actual processing of requests. NGINX relies on OS-dependent mechanisms to efficiently distribute requests among worker processes. The number of worker processes is defined by the [worker_processes](https://nginx.org/en/docs/ngx_core_module.html#worker_processes) directive in the **nginx.conf** configuration file and can either be set to a fixed number or configured to adjust automatically to the number of available CPU cores. ## Control NGINX with signals {#signals} To reload your configuration, you can stop or restart NGINX, or send signals to the master process. A signal can be sent by running the `nginx` command (invoking the NGINX executable) with the `-s` argument. ```none nginx -s ``` where `` can be one of the following: - `quit` – Shut down gracefully (the `SIGQUIT` signal) - `reload` – Reload the configuration file (the `SIGHUP` signal) - `reopen` – Reopen log files (the `SIGUSR1` signal) - `stop` – Shut down immediately (or fast shutdown, the `SIGTERM` signal) The `kill` utility can also be used to send a signal directly to the master process. The process ID of the master process is written, by default, to the **nginx.pid** file, which is located in the **/usr/local/nginx/logs** or **/var/run** directory. For more information about advanced signals (for performing live binary upgrades, for example), see [Control nginx](https://nginx.org/en/docs/control.html) at **nginx.org**. ## Control NGINX with Control API {#control-api} In addition to [signal-based controls](#signals), NGINX Plus and NGINX Open Source can be controlled with the Control REST API, available since [NGINX Plus PLS R37.0.0 LTS](nginx/releases.md#r37.0) and NGINX Open Source 1.31.5. The Control API is implemented in the [NGINX master process](#processes) and provides a REST interface for runtime control and inspection. It allows you to: - view worker process information (process name, PID, and exit state); - get a memory dump of loaded NGINX configuration files; - trigger a configuration reload similar to `nginx -s reload` and review reload logs. The Control API provides the following endpoints: - `/1/control/processes` — return worker process information; - `/1/control/config` — return the in-memory NGINX configuration and trigger reloads with `PATCH`; - `/1/nginx` — return NGINX version and build information. See [NGINX Control REST API reference](nginx/admin-guide/basic-functionality/control-api-reference.md) for details. ### Enable NGINX Control API By default, the Control API is disabled. To enable it, start NGINX Open Source or NGINX Plus with the [`-l` argument](https://nginx.org/en/docs/switches.html#l), specifying a UNIX-domain socket or TCP port: ```shell sudo nginx -l unix:/tmp/nginx.sock ``` For NGINX Open Source 1.31.5 and later, make sure NIGNX is built with the `--with-control-api` configuration parameter: ```shell nginx -V 2>&1 | grep -- '--with-control-api' ``` If the parameter is enabled, the command highlights the `--with-control-api` parameter in the nginx build configuration. ### Security implications - Never expose the Control API to the public Internet. Restrict access to the Control API listener with firewalls or ACLs, where possible, place it on a dedicated interface or VLAN. - Configure the Control API to listen on a UNIX-domain socket. This is currently the most effective way to control access, because authorization can be managed through file permissions, and the created socket file is accessible only to the user running NGINX. - Keep your NGINX binary and the operating system up to date. ### Control API reference documentation See [NGINX Control REST API reference](nginx/admin-guide/basic-functionality/control-api-reference.md) for details on available endpoints, request parameters, and response schemas. [[icon: download]Download Control API OpenAPI YAML specification](/nginx/admin-guide/yaml/nginx-control-api/1/nginx_control_api.yaml) or copy this link to explore the API using standard OpenAPI-compatible tools such as Redoc or Swagger UI. -------------------------------------------------------------------------------- title: "Installing NGINX Open Source" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/index.md description: Install NGINX Open Source either as a prebuilt package or from source, following step-by-step instructions for all supported Linux distributions. -------------------------------------------------------------------------------- This article explains how to install NGINX Open Source on various operating systems, including an overview of existing NGINX Open Source versions, installation types and methods, modules included in the default package and dynamic modules packages, and the basics of compiling NGINX from the source code. ## Stable and mainline versions NGINX Open Source is available in two versions: Mainline and Stable. The source code and release notes for both versions are available from [NGINX download page](https://nginx.org/en/download.html). ### Mainline version Mainline version (also Mainline release, Mainline branch) is the latest development version, updated approximately every 1 or 2 months, includes the latest features, bug fixes and security fixes. This version is recommended for production unless your organization has strict requirements for stability, in which case the stable version might be the better choice. The Mainline version always has an odd middle number, for example, 1.*27*.X. ### Stable version Stable version (also Stable release, Stable branch) is updated typically once a year or as needed for critical bug fixes or security fixes that are always backported from the Mainline version. This version is recommended for environments with strict requirements for stability. The Stable version is always even-numbered, for example, 1.*28*.X. ## Distribution and installation methods Both the NGINX Open Source Mainline and Stable versions can be obtained and installed in several ways: - A package from the [official NGINX Open Source repository](#official-repository) (recommended for production). This is the most reliable method: you have to set up the repository once, but after that the provided package is always up to date. - A package from your [operating system’s default package repository](#os-default-repository) (suitable for learning or testing). This is the easiest method, but generally the provided package is outdated. - Your own package [compiled from source](#sources) (recommended for advanced and custom builds). This method is the most flexible: you can include non-standard or third‑party modules, apply the latest security patches, or build the binary for almost any Unix-like operating system using different compilers and custom compiler options. - A container: suitable for development, testing, production deployments in container environments such as Docker, Podman, Kubernetes: see [Deploying NGINX with Docker](/nginx/admin-guide/installing-nginx/installing-nginx-docker.md#using-nginx-open-source-docker-images) ## OS default repository Installing from your operating system’s default repository is the simplest and fastest method. It is suitable for demos, learning, or testing environments. However, for production use, there are some considerations: - The package is maintained by the distribution’s repository maintainers, so F5/NGINX cannot verify its contents, build process, or update frequency. - The package may be outdated. To compare release versions and features, see the [Changelog](https://nginx.org/en/CHANGES). - Some Linux distros, especially those focused on long-term support, provide the Stable version only. For the latest official release and better control over updates, it is recommended to configure your package manager to install [from the official NGINX repository](#official-repository). The installation steps include updating the package repository and installing the `nginx` binary. - For MacOS: ```shell # ensure Homebrew is installed, see https://brew.sh/ brew update && \ brew install nginx ``` - For RHEL-based distributions and Amazon Linux 2023: ```shell sudo dnf update -y && \ sudo dnf install nginx ``` If the output of the command is `no nginx package found`, install the [EPEL repository](https://docs.fedoraproject.org/en-US/epel/) that contains the nginx package and run the command again. The EPEL installation steps include installing the repository, clearing the packages cache and updating the repository info: ```shell sudo dnf update -y && \ sudo dnf install epel-release -y && \ sudo dnf clean all && \ sudo dnf update -y ``` - For Debian-based distributions: ```shell sudo apt update -y && \ sudo apt install nginx ``` - For FreeBSD or DragonFly BSD: ```shell sudo pkg update -y && \ sudo pkg install nginx ``` - For SUSE Linux Enterprise or openSUSE: ```shell # ensure Web-Scripting-Module repo is enabled sudo zypper refresh -y && \ sudo zypper install nginx ``` - For Alpine: ```shell # ensure community repo is enabled in /etc/apk/repositories sudo apk update && \ sudo apk add nginx ``` - For Amazon Linux 2: ```shell sudo yum install -y amazon-linux-extras amazon-linux-extras list | grep nginx # get package name, e.g. nginx1 sudo amazon-linux-extras install nginx1 ``` Check the version installed: ```shell sudo nginx -v ``` It is recommended to review the [Changelog](https://nginx.org/en/CHANGES) and compare the features, bugfixes, and security fixes between the installed version and the latest release. The latest NGINX Open Source version is always available from [the official repository](#official-repository). You can find out the path to NGINX configuration file, error log, and access log files using the command: ```shell nginx -V 2>&1 | awk -F: '/configure arguments/ {print $2}' | xargs -n1 ``` where: - `--conf-path=` is the path to the `nginx.conf` configuration file - `--error-log-path=` is the path to the error log - `--http-log-path=` is the path to the access log ## Official repository You can configure your package manager to install NGINX Open Source from the official **nginx.org** repository, which provides both the latest [Mainline and Stable](#stable_vs_mainline) versions for most major production operating systems. See the [Changelog](https://nginx.org/en/CHANGES) for version details. You need to set up the repository once, but after that the provided package will stay up to date with the latest releases. Before installing, ensure your operating system and architecture are [supported](https://nginx.org/en/linux_packages.html#distributions). This installation method is recommended for production environments. ### Repository contents The repository contains the latest versions of the following packages: | Package name | Description | | ---------------------------| ---------------------------------------------| | `nginx` | Main NGINX Open Source package, contains [core modules](#prebuilt_modules) and [statically linked modules](#statically-linked-modules). | | `nginx-module-geoip` | The [`ngx_http_geoip_module`](https://nginx.org/en/docs/http/ngx_http_geoip_module.html) as a [dynamic module](#dynamic-modules). | | `nginx-module-image-filter`| The [`ngx_http_image-filter_module`](https://nginx.org/en/docs/http/ngx_http_image_filter_module.html) as a [dynamic module](#dynamic-modules). | | `nginx-module-njs` | The [`ngx_http_js_module`](https://nginx.org/en/docs/http/ngx_http_js_module.html) and [`ngx_stream_js_module`](https://nginx.org/en/docs/stream/ngx_stream_js_module.html) as [dynamic modules](#dynamic-modules). | | `nginx-module-perl` | The [`ngx_http_perl_module`](https://nginx.org/en/docs/http/ngx_http_perl_module.html) as a [dynamic module](#dynamic-modules). | | `nginx-module-xslt` | The [`ngx_http_xsl_module`](https://nginx.org/en/docs/http/ngx_http_xslt_module.html) as a [dynamic module](#dynamic-modules). | | `nginx-module-otel` | The [`ngx_otel_module`](https://nginx.org/en/docs/ngx_otel_module.html) as a [dynamic module](#dynamic-modules). | | `nginx-module-acme` | The [`ngx_http_acme_module`](https://nginx.org/en/docs/http/ngx_http_acme_module.html) as a [dynamic module](#dynamic-modules). | ### RHEL-based packages RHEL-based operating systems include RHEL, CentOS, Oracle Linux, AlmaLinux, Rocky Linux. Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Set up the `yum` or `dnf` repository. - In the `/etc/yum.repos.d` directory, create the `nginx.repo` file using any text editor, for example, `vi`: ```shell sudo vi /etc/yum.repos.d/nginx.repo ``` - Add the following lines to the file, where `nginx-stable` and `nginx-mainline` point to the latest Stable or Mainline version of NGINX Open Source: ```text [nginx-stable] name=nginx stable repo baseurl=https://nginx.org/packages/centos/$releasever/$basearch/ gpgcheck=1 enabled=1 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true [nginx-mainline] name=nginx mainline repo baseurl=https://nginx.org/packages/mainline/centos/$releasever/$basearch/ gpgcheck=1 enabled=0 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true ``` - Save the file. 4. If needed, switch to `nginx-mainline` packages instead of `nginx-stable` that are enabled by default (the `enable=1` parameter): ```shell sudo dnf config-manager --enable nginx-mainline ``` 5. Update the repository: ```shell sudo dnf update ``` 6. Install the `nginx` package: ```shell sudo dnf install nginx ``` When prompted to accept the GPG key, verify that the following three fingerprints match, and if so, accept them: ```shell Importing GPG key 0xB49F6B46: UserID : "nginx signing key ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib/nginx/modules` directory. ### Debian packages This section covers Debian packages only. For Ubuntu-specific instructions, see [Ubuntu packages](#ubuntu-packages). Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Install the prerequisites: ```shell sudo apt update && \ sudo apt install curl \ gnupg2 \ ca-certificates \ lsb-release \ debian-archive-keyring ``` 2. Import an official nginx signing key to allow `apt` to verify the authenticity of packages. Fetch the key: ```shell curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor \ | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null ``` 3. Verify that the downloaded file contains the correct signing key: ```shell gpg --dry-run --quiet --no-keyring --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg ``` The output should list the following three full fingerprints: ```none pub rsa4096 2024-05-29 [SC] 8540A6F18833A80E9C1653A42FD21310B49F6B46 uid nginx signing key pub rsa2048 2011-08-19 [SC] [expires: 2027-05-24] 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 uid nginx signing key pub rsa4096 2024-05-29 [SC] 9E9BE90EACBCDE69FE9B204CBCDCD8A38D88A2B3 uid nginx signing key ``` If the fingerprints do not match, delete the file immediately. 4. Set up the `apt` repository to fetch packages from either `stable` or `mainline` branch. - For `stable`: ```shell echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://nginx.org/packages/debian `lsb_release -cs` nginx" \ | sudo tee /etc/apt/sources.list.d/nginx.list ``` - For `mainline`: ```shell echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://nginx.org/packages/mainline/debian `lsb_release -cs` nginx" \ | sudo tee /etc/apt/sources.list.d/nginx.list ``` 5. Set up repository pinning to prioritize official nginx packages over those provided by the distribution: ```shell echo -e "Package: *\nPin: origin nginx.org\nPin: release o=nginx\nPin-Priority: 900\n" \ | sudo tee /etc/apt/preferences.d/99nginx ``` 6. Install the `nginx` package: ```shell sudo apt update && \ sudo apt install nginx ``` 7. Start NGINX Open Source: ```shell sudo nginx ``` 8. Verify that NGINX Open Source is up and running using the `curl` command: ```shell curl -I 127.0.0.1 ``` Expected output: ```shell HTTP/1.1 200 OK Server: nginx/1.29.8 ``` After installation, the following files are available for configuration and troubleshooting: - Configuration file: `nginx.conf`, located in `/etc/nginx/` - Log files: `access.log` and `error.log`, located in `/var/log/nginx/` 9. If needed, install one or more [dynamic module packages](#repository-contents): ```shell sudo apt install nginx-module- ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib/nginx/modules` directory. ### Ubuntu packages Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Install the prerequisites: ```shell sudo apt update && \ sudo apt install curl \ gnupg2 \ ca-certificates \ lsb-release \ ubuntu-keyring ``` 2. Import an official nginx signing key to allow `apt` to verify the authenticity of packages. Fetch the key: ```shell curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor \ | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null ``` 3. Verify that the downloaded file contains the correct signing key: ```shell gpg --dry-run --quiet --no-keyring --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg ``` The output should list the following three full fingerprints: ```none pub rsa4096 2024-05-29 [SC] 8540A6F18833A80E9C1653A42FD21310B49F6B46 uid nginx signing key pub rsa2048 2011-08-19 [SC] [expires: 2027-05-24] 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 uid nginx signing key pub rsa4096 2024-05-29 [SC] 9E9BE90EACBCDE69FE9B204CBCDCD8A38D88A2B3 uid nginx signing key ``` If the fingerprints do not match, delete the file immediately. 4. Set up the `apt` repository to fetch packages from either `stable` or `mainline` branch. - For `stable`: ```shell echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \ | sudo tee /etc/apt/sources.list.d/nginx.list ``` - For `mainline`: ```shell echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://nginx.org/packages/mainline/ubuntu `lsb_release -cs` nginx" \ | sudo tee /etc/apt/sources.list.d/nginx.list ``` 5. Set up repository pinning to prioritize official nginx packages over those provided by the distribution: ```shell echo -e "Package: *\nPin: origin nginx.org\nPin: release o=nginx\nPin-Priority: 900\n" \ | sudo tee /etc/apt/preferences.d/99nginx ``` 6. Install the `nginx` package: ```shell sudo apt update && \ sudo apt install nginx ``` 7. Start NGINX Open Source: ```shell sudo nginx ``` 8. Verify that NGINX Open Source is up and running using the `curl` command: ```shell curl -I 127.0.0.1 ``` Expected output: ```shell HTTP/1.1 200 OK Server: nginx/1.29.8 ``` After installation, the following files are available for configuration and troubleshooting: - Configuration file: `nginx.conf`, located in `/etc/nginx/` - Log files: `access.log` and `error.log`, located in `/var/log/nginx/` 9. If needed, install one or more [dynamic module packages](#repository-contents): ```shell sudo apt install nginx-module- ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib/nginx/modules` directory. ### SUSE Linux Enterprise packages Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Install the prerequisites: ```shell sudo zypper refresh && \ sudo zypper install curl \ ca-certificates \ gpg2 ``` 2. Set up the `zypper` repository to fetch packages from either `stable` or `mainline` branch. - For `stable`: ```shell sudo zypper addrepo --gpgcheck --type yum --refresh --check \ 'https://nginx.org/packages/sles/$releasever_major' nginx-stable ``` - For `mainline`: ```shell sudo zypper addrepo --gpgcheck --type yum --refresh --check \ 'https://nginx.org/packages/mainline/sles/$releasever_major' nginx-mainline ``` 3. Import the official nginx signing key so `zypper` and `rpm` could verify the packages authenticity. Fetch the key: ```shell curl -o /tmp/nginx_signing.key https://nginx.org/keys/nginx_signing.key ``` 4. Verify that the downloaded file contains the proper key: ```shell gpg --show-keys /tmp/nginx_signing.key ``` The output should list the following three full fingerprints: ```none pub rsa4096 2024-05-29 [SC] 8540A6F18833A80E9C1653A42FD21310B49F6B46 uid nginx signing key pub rsa2048 2011-08-19 [SC] [expires: 2027-05-24] 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 uid nginx signing key pub rsa4096 2024-05-29 [SC] 9E9BE90EACBCDE69FE9B204CBCDCD8A38D88A2B3 uid nginx signing key ``` If the fingerprints do not match, delete the file immediately. 5. Import the key to the `rpm` database: ```shell sudo rpmkeys --import /tmp/nginx_signing.key ``` 6. Install the `nginx` package: ```shell sudo zypper install nginx ``` 7. Verify that NGINX Open Source is up and running using the `curl` command: ```shell curl -I 127.0.0.1 ``` Expected output: ```shell HTTP/1.1 200 OK Server: nginx/1.29.8 ``` After installation, the following files are available for configuration and troubleshooting: - Configuration file: `nginx.conf`, located in `/etc/nginx/` - Log files: `access.log` and `error.log`, located in `/var/log/nginx/` 8. If needed, install one or more [dynamic module packages](#repository-contents): ```shell sudo zypper install nginx-module- ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib64/nginx/modules` directory. ### Alpine Linux packages Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Install the prerequisites: ```shell sudo apk add openssl \ curl \ ca-certificates ``` 2. Set up the `apk` repository to fetch packages from either `stable` or `mainline` branch by adding the repository URL to the `/etc/apk/repositories` file and the `@nginx` tag. - For `stable`: ```shell printf "%s%s%s%s\n" \ "@nginx " \ "https://nginx.org/packages/alpine/v" \ `egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release` \ "/main" \ | sudo tee -a /etc/apk/repositories ``` - For `mainline`: ```shell printf "%s%s%s%s\n" \ "@nginx " \ "https://nginx.org/packages/mainline/alpine/v" \ `egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release` \ "/main" \ | sudo tee -a /etc/apk/repositories ``` 3. Import the official nginx signing key so that `apk` can verify the authenticity of packages. Fetch the key: ```shell curl -o /tmp/nginx_signing.rsa.pub https://nginx.org/keys/nginx_signing.rsa.pub ``` 4. Verify that the downloaded file contains the proper public key: ```shell openssl rsa -pubin -in /tmp/nginx_signing.rsa.pub -text -noout ``` The output should contain the following modulus: ```none Public-Key: (2048 bit) Modulus: 00:fe:14:f6:0a:1a:b8:86:19:fe:cd:ab:02:9f:58: 2f:37:70:15:74:d6:06:9b:81:55:90:99:96:cc:70: 5c:de:5b:e8:4c:b2:0c:47:5b:a8:a2:98:3d:11:b1: f6:7d:a0:46:df:24:23:c6:d0:24:52:67:ba:69:ab: 9a:4a:6a:66:2c:db:e1:09:f1:0d:b2:b0:e1:47:1f: 0a:46:ac:0d:82:f3:3c:8d:02:ce:08:43:19:d9:64: 86:c4:4e:07:12:c0:5b:43:ba:7d:17:8a:a3:f0:3d: 98:32:b9:75:66:f4:f0:1b:2d:94:5b:7c:1c:e6:f3: 04:7f:dd:25:b2:82:a6:41:04:b7:50:93:94:c4:7c: 34:7e:12:7c:bf:33:54:55:47:8c:42:94:40:8e:34: 5f:54:04:1d:9e:8c:57:48:d4:b0:f8:e4:03:db:3f: 68:6c:37:fa:62:14:1c:94:d6:de:f2:2b:68:29:17: 24:6d:f7:b5:b3:18:79:fd:31:5e:7f:4c:be:c0:99: 13:cc:e2:97:2b:dc:96:9c:9a:d0:a7:c5:77:82:67: c9:cb:a9:e7:68:4a:e1:c5:ba:1c:32:0e:79:40:6e: ef:08:d7:a3:b9:5d:1a:df:ce:1a:c7:44:91:4c:d4: 99:c8:88:69:b3:66:2e:b3:06:f1:f4:22:d7:f2:5f: ab:6d Exponent: 65537 (0x10001) ``` 5. Move the key to `apk` trusted keys storage: ```shell sudo mv /tmp/nginx_signing.rsa.pub /etc/apk/keys/ ``` 6. Install the `nginx` package: ```shell sudo apk add nginx@nginx ``` 7. Start NGINX Open Source: ```shell sudo nginx ``` 8. Verify that NGINX Open Source is up and running using the `curl` command: ```shell curl -I 127.0.0.1 ``` Expected output: ```shell HTTP/1.1 200 OK Server: nginx/1.29.8 ``` After installation, the following files are available for configuration and troubleshooting: - Configuration file: `nginx.conf`, located in `/etc/nginx/` - Log files: `access.log` and `error.log`, located in `/var/log/nginx/` 9. If needed, install one or more [dynamic module packages](#repository-contents). The `@nginx` tag should also be specified: ```shell sudo apk add nginx-module-@nginx nginx-module-@nginx ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib/nginx/modules` directory. ### Amazon Linux 2 packages Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Install the prerequisites: ```shell sudo yum install yum-utils ``` 2. Set up the `yum` repository. - In the `/etc/yum.repos.d` directory, create the `nginx.repo` file using any text editor, for example, `vi`: ```shell sudo vi /etc/yum.repos.d/nginx.repo ``` - Add the following lines to file, where the `nginx-stable` and `nginx-mainline` elements point to the latest stable or mainline version of NGINX Open Source: ```none [nginx-stable] name=nginx stable repo baseurl=https://nginx.org/packages/amzn2/$releasever/$basearch/ gpgcheck=1 enabled=1 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true [nginx-mainline] name=nginx mainline repo baseurl=https://nginx.org/packages/mainline/amzn2/$releasever/$basearch/ gpgcheck=1 enabled=0 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true ``` - Save the file. 3. If needed, switch to `nginx-mainline` packages instead of `nginx-stable` that are enabled by default (the `enable=1` parameter): ```shell sudo yum-config-manager --enable nginx-mainline ``` 4. Update the repository: ```shell sudo yum update ``` 5. Install nginx: ```shell sudo yum install nginx ``` When prompted to accept the GPG key, verify that the following three fingerprints match: `8540 A6F1 8833 A80E 9C16 53A4 2FD2 1310 B49F 6B46`, `573B FD6B 3D8F BC64 1079 A6AB ABF5 BD82 7BD9 BF62`, `9E9B E90E ACBC DE69 FE9B 204C BCDC D8A3 8D88 A2B3`, and if so, accept them. 6. Start NGINX Open Source: ```shell sudo nginx ``` 7. Verify that NGINX Open Source is up and running using the `curl` command: ```shell curl -I 127.0.0.1 ``` Expected output: ```shell HTTP/1.1 200 OK Server: nginx/1.29.8 ``` After installation, the following files are available for configuration and troubleshooting: - Configuration file: `nginx.conf`, located in `/etc/nginx/` - Log files: `access.log` and `error.log`, located in `/var/log/nginx/` 8. If needed, install one or more [dynamic module packages](#repository-contents): ```shell sudo yum install nginx-module- ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib64/nginx/modules` directory. ### Amazon Linux 2023 packages Before installing, check if your operating system and architecture are supported, see [Supported distributions and versions](https://nginx.org/en/linux_packages.html#distributions). 1. Install the prerequisites: ```shell sudo yum install yum-utils ``` 2. To set up the `yum` repository for Amazon Linux 2023, create the file named `/etc/yum.repos.d/nginx.repo` with the following contents: ```none [nginx-stable] name=nginx stable repo baseurl=https://nginx.org/packages/amzn/2023/$basearch/ gpgcheck=1 enabled=1 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true [nginx-mainline] name=nginx mainline repo baseurl=https://nginx.org/packages/mainline/amzn/2023/$basearch/ gpgcheck=1 enabled=0 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true ``` By default, the repository for `stable` nginx packages is used. If you would like to use `mainline` nginx packages, run the following command: ```shell sudo yum-config-manager --enable nginx-mainline ``` 3. Install nginx: ```shell sudo yum install nginx ``` When prompted to accept the GPG key, verify that the following three fingerprints match: `8540 A6F1 8833 A80E 9C16 53A4 2FD2 1310 B49F 6B46`, `573B FD6B 3D8F BC64 1079 A6AB ABF5 BD82 7BD9 BF62`, `9E9B E90E ACBC DE69 FE9B 204C BCDC D8A3 8D88 A2B3` and if so, accept them. 4. Start NGINX Open Source: ```shell sudo nginx ``` 5. Verify that NGINX Open Source is up and running using the `curl` command: ```shell curl -I 127.0.0.1 ``` Expected output: ```shell HTTP/1.1 200 OK Server: nginx/1.29.8 ``` After installation, the following files are available for configuration and troubleshooting: - Configuration file: `nginx.conf`, located in `/etc/nginx/` - Log files: `access.log` and `error.log`, located in `/var/log/nginx/` 6. If needed, install one or more [dynamic module packages](#repository-contents): ```shell sudo dnf install nginx-module- ``` Then, enable each module in the `nginx.conf` configuration file using the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive. The resulting `.so` files are located in the `/usr/lib64/nginx/modules` directory. ## Package contents NGINX has a modular architecture that allows functionality to be included selectively. Some modules are built into the core and are always included in the package. Others are not part of the core, but are compiled at build time using the `--with-` configuration flag. Additionally, some modules, usually those with external dependencies, are distributed as separate packages that can be loaded at runtime as [dynamic modules](#dynamic-modules). Packaging sources and scripts can be found in the [packaging sources repository](https://github.com/nginx/pkg-oss). ### Core modules NGINX core modules are built-in components that provide essential functionality such as configuration parsing, event handling, process management, and HTTP request processing. They are statically compiled into the NGINX binary and cannot be disabled or excluded. |Module Name | Description | | -------------------------| ---------------------------------------------| | [`ngx_core_module`](https://nginx.org/en/docs/ngx_core_module.html) | Internal core functionality. | | [`ngx_http_core_module`](https://nginx.org/en/docs/http/ngx_http_core_module.html) | Essential HTTP functionality ([`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location), [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server), [`listen`](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen)). | | [`ngx_http_access_module`](https://nginx.org/en/docs/http/ngx_http_access_module.html)| Access control ([`allow`](https://nginx.org/en/docs/http/ngx_http_access_module.html#allow), [`deny`](https://nginx.org/en/docs/http/ngx_http_access_module.html#deny)). | | [`ngx_http_auth_basic_module`](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) | HTTP Basic Auth. | | [`ngx_http_autoindex_module`](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html) |Directory listing. | | [`ngx_http_browser_module`](https://nginx.org/en/docs/http/ngx_http_browser_module.html) | Browser-based conditional logic.| | [`ngx_http_charset_module`](https://nginx.org/en/docs/http/ngx_http_charset_module.html) | Character set conversion.| | [`ngx_http_empty_gif_module`](https://nginx.org/en/docs/http/ngx_http_empty_gif_module.html) | Serves a 1x1 transparent GIF.| | [`ngx_http_fastcgi_module`](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html) | FastCGI backend support. | | [`ngx_http_geo_module`](https://nginx.org/en/docs/http/ngx_http_geo_module.html) | IP-based variable creation. | | [`ngx_http_gzip_module`](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) | Gzip compression. | | [`ngx_http_headers_module`](https://nginx.org/en/docs/http/ngx_http_headers_module.html) | Add/modify response headers.| | [`ngx_http_index_module`](https://nginx.org/en/docs/http/ngx_http_index_module.html) | Default index file (e.g. index.html). | | [`ngx_http_limit_conn_module`](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html) | Limits concurrent connections. | | [`ngx_http_limit_req_module`](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html) | Limits request rate. | | [`ngx_http_log_module`](https://nginx.org/en/docs/http/ngx_http_log_module.html) | Logging support. | | [`ngx_http_map_module`](https://nginx.org/en/docs/http/ngx_http_map_module.html) | Variable mapping logic. | | [`ngx_http_memcached_module`](https://nginx.org/en/docs/http/ngx_http_memcached_module.html) | Memcached backend. | | [`ngx_http_mirror_module`](https://nginx.org/en/docs/http/ngx_http_mirror_module.html) | Mirrors requests to another location. | | [`ngx_http_proxy_module`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) | HTTP reverse proxy. | | [`ngx_http_referer_module`](https://nginx.org/en/docs/http/ngx_http_referer_module.html) | Referer-based access control.| | [`ngx_http_rewrite_module`](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html) | URL rewriting. | | [`ngx_http_scgi_module`](https://nginx.org/en/docs/http/ngx_http_scgi_module.html) | SCGI backend. | | [`ngx_http_split_clients_module`](https://nginx.org/en/docs/http/ngx_http_split_clients_module.html) | A/B testing logic. | | [`ngx_http_ssi_module`](https://nginx.org/en/docs/http/ngx_http_ssi_module.html) | Server Side Includes. | | [`ngx_http_upstream_module`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html) | Upstream group and load balancing logic. | | [`ngx_http_userid_module`](https://nginx.org/en/docs/http/ngx_http_userid_module.html) | Cookie-based user ID generation. | | [`ngx_http_uwsgi_module`](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html) | uWSGI backend. | ### Statically linked modules In addition to core modules, the `nginx` package includes other nginx modules that do not require additional libraries to avoid extra dependencies. These modules are compiled at build time and specified using the` --with-` configure option. Unlike [dynamic modules](#dynamic-modules), they cannot be enabled or disabled dynamically after compilation. You can get the list of these modules in the output of the `nginx -V` command output in `configure arguments`. |Module Name | Description | | -------------------------| ---------------------------------------------| | `--with-compat` |Enables dynamic modules compatibility. | | `--with-file-aio` |Enables the use of [asynchronous file I/O](https://nginx.org/en/docs/http/ngx_http_core_module.html#aio) (AIO) on FreeBSD and Linux. | | `--with-threads` | Enables NGINX to use thread pools. For details, see [Thread Pools in NGINX Boost Performance 9x!](https://www.nginx.com/blog/thread-pools-boost-performance-9x/) on the NGINX blog. | |[`--with-http_addition_module`](https://nginx.org/en/docs/http/ngx_http_addition_module.html)| Adds text before and after a response. | |[`--with-http_auth_request_module`](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html)|Implements client authorization based on the result of a subrequest. | |[`--with-http_dav_module`](https://nginx.org/en/docs/http/ngx_http_dav_module.html)|Enables file management automation using the WebDAV protocol. | |[`--with-http_flv_module`](https://nginx.org/en/docs/http/ngx_http_flv_module.html)|Provides pseudo-streaming server-side support for Flash Video (FLV) files. | |[`--with-http_gunzip_module`](https://nginx.org/en/docs/http/ngx_http_gunzip_module.html)|Decompresses responses with `Content-Encoding: gzip` for clients that do not support the _zip_ encoding method. | |[`--with-http_gzip_static_module`](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html)| Allows sending precompressed files with the **.gz** filename extension instead of regular files. | |[`--with-http_mp4_module`](https://nginx.org/en/docs/http/ngx_http_mp4_module.html)| Provides pseudo-streaming server-side support for MP4 files. | |[`--with-http_random_index_module`](https://nginx.org/en/docs/http/ngx_http_random_index_module.html) | Processes requests ending with the slash character (‘/’) and picks a random file in a directory to serve as an index file. | |[`--with-http_realip_module`](https://nginx.org/en/docs/http/ngx_http_realip_module.html) | Changes the client address to the one sent in the specified header field. | |[`--with-http_secure_link_module`](https://nginx.org/en/docs/http/ngx_http_secure_link_module.html) | Used to check authenticity of requested links, protect resources from unauthorized access, and limit link lifetime. | |[`--with-http_slice_module`](https://nginx.org/en/docs/http/ngx_http_slice_module.html) | Allows splitting a request into subrequests, each subrequest returns a certain range of response. Provides more effective caching of large files. | |[`--with-http_ssl_module`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) | Enables HTTPS support. Requires an SSL library such as [OpenSSL](https://www.openssl.org/). | |[`--with-http_stub_status_module`](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html)| Provides access to basic status information. Note that NGINX Plus customers do not require this module as they are already provided with extended status metrics and interactive dashboard. | |[`--with-http_sub_module`](https://nginx.org/en/docs/http/ngx_http_sub_module.html) | Modifies a response by replacing one specified string by another. | |[`--with-http_v2_module`](https://nginx.org/en/docs/http/ngx_http_v2_module.html)| Enable support for [HTTP/2](https://datatracker.ietf.org/doc/html/rfc7540). See [The HTTP/2 Module in NGINX](https://www.nginx.com/blog/http2-module-nginx/) on the NGINX blog for details.| |[`--with-http_v3_module`](https://nginx.org/en/docs/http/ngx_http_v3_module.html)| Provides experimental support for [HTTP/3](https://datatracker.ietf.org/doc/html/rfc9114). | |[`--with-mail`](https://nginx.org/en/docs/mail/ngx_mail_core_module.html)| Enables mail proxy functionality. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-mail=dynamic`. | |[`--with-mail_ssl_module`](https://nginx.org/en/docs/mail/ngx_mail_ssl_module.html)| Provides support for a mail proxy server to work with the SSL/TLS protocol. Requires an SSL library such as [OpenSSL](https://www.openssl.org/). | |[`--with-stream`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html) | Enables the TCP and UDP proxy functionality. To compile as a separate [dynamic module](nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-stream=dynamic`. | |[`--with-stream_realip_module`](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html) | Changes the client address and port to the ones sent in the PROXY protocol header. | |[`--with-stream_ssl_module`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html)| Provides support for a stream proxy server to work with the SSL/TLS protocol. Requires an SSL library such as [OpenSSL](https://www.openssl.org/). | ## Dynamic modules Some modules, especially those with external dependencies, are not included in the `nginx` package. However, they are available as separate packages from the official NGINX repository. After installation they can be connected as a dynamic modules, or shared object (.so) files via the `load_module` directive in the NGINX configuration. |Module Name | Description | Package name | | -------------------------| ---------------------------------------------|---------------| | [`ngx_http_geoip_module`](https://nginx.org/en/docs/http/ngx_http_geoip_module.html) | Creates variables with values depending on the client IP address, using the precompiled MaxMind databases. | `nginx-module-geoip` | | [`ngx_http_image-filter_module`](https://nginx.org/en/docs/http/ngx_http_image_filter_module.html)| Transforms images in JPEG, GIF, PNG, and WebP formats. | `nginx-module-image-filter` | | [`ngx_http_js_module`](https://nginx.org/en/docs/http/ngx_http_js_module.html), [`ngx_stream_js_module`](https://nginx.org/en/docs/stream/ngx_stream_js_module.html)| Extends the server's functionality through JavaScript scripting, enabling the creation of custom server-side logic. | `nginx-module-njs` | | [`ngx_http_perl_module`](https://nginx.org/en/docs/http/ngx_http_perl_module.html) | Implements location and variable handlers in Perl and inserts Perl calls into SSI. | `nginx-module-perl` | | [`ngx_http_xsl_module`](https://nginx.org/en/docs/http/ngx_http_xslt_module.html) | Transforms XML responses using one or more XSLT stylesheets. | `nginx-module-xslt` | | [`ngx_otel_module`](https://nginx.org/en/docs/ngx_otel_module.html) | Provides OpenTelemetry distributed tracing support. | `nginx-module-otel` | | [`ngx_http_acme_module`](https://nginx.org/en/docs/http/ngx_http_acme_module.html) | Implements the automatic certificate management (ACMEv2) protocol. | `nginx-module-acme` | ## Compile and install from source Compiling from source affords more flexibility than using prebuilt packages: you can add specific modules, both official and third party, and apply the latest security patches. ### Install NGINX dependencies Prior to compiling NGINX Open Source from source, you need to install libraries for its dependencies: - [PCRE](http://pcre.org/) – Supports regular expressions. Required by the NGINX [Core](https://nginx.org/en/docs/ngx_core_module.html) and [Rewrite](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html) modules. ```shell wget https://github.com/PCRE2Project/pcre2/releases/download/pcre2-10.43/pcre2-10.43.tar.gz && \ tar -zxf pcre2-10.43.tar.gz && \ cd pcre2-10.43 && \ ./configure && \ make && \ sudo make install ``` - [zlib](http://www.zlib.net/) – Supports header compression. Required by the NGINX [Gzip](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) module. ```shell wget http://zlib.net/zlib-1.3.1.tar.gz && \ tar -zxf zlib-1.3.1.tar.gz && \ cd zlib-1.3.1 && \ ./configure && \ make && \ sudo make install ``` - [OpenSSL](https://www.openssl.org/) – Supports the HTTPS protocol. Required by the NGINX [SSL](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) module and others. ```shell wget https://www.openssl.org/source/openssl-3.0.13.tar.gz #for Ubuntu and Debian # curl -LO https://www.openssl.org/source/openssl-3.0.13.tar.gz #for RHEL-based: tar -zxf openssl-3.0.13.tar.gz cd openssl-3.0.13 ./config --prefix=/usr/local --openssldir=/usr/local/ssl make -j$(nproc) sudo make install ``` ### Download the sources {#sources_download} Download the source files for both the stable and mainline versions from [**nginx.org**](https://www.nginx.org/en/download.html). To download and unpack the source for the latest _mainline_ version, run: ```shell wget https://nginx.org/download/nginx-1.29.8.tar.gz && \ tar zxf nginx-1.29.8.tar.gz && \ cd nginx-1.29.8 ``` To download and unpack source files for the latest _stable_ version, run: ```shell wget https://nginx.org/download/nginx-1.30.0.tar.gz && \ tar zxf nginx-1.30.0.tar.gz && \ cd nginx-1.30.0 ``` ### Configure the build options {#configure} Configure options are specified with the `./configure` script that sets up various NGINX parameters, including paths to source and configuration files, compiler options, connection processing methods, and the list of modules. The script finishes by creating the `Makefile` required to compile the code and install NGINX Open Source. An example of options to the `configure` script: ```shell ./configure \ --sbin-path=/usr/local/nginx/nginx \ --conf-path=/usr/local/nginx/nginx.conf \ --pid-path=/usr/local/nginx/nginx.pid \ --with-pcre=../pcre2-10.43 \ --with-zlib=../zlib-1.3.1 \ --with-http_ssl_module \ --with-stream \ --with-mail=dynamic \ --add-module=/usr/build/nginx-rtmp-module \ --add-dynamic-module=/usr/build/3party_module ``` ### Configure NGINX paths {#configure_paths} The `configure` script allows you to set paths to NGINX binary and configuration files, and to dependent libraries such as PCRE or SSL, in order to link them statically to the NGINX binary. |Parameter | Description | | ---| --- | |`--prefix=` | Directory for NGINX files, and the base location for all relative paths set by the other `configure` script options (excluding paths to libraries) and for the path to the **nginx.conf** configuration file. Default: **/usr/local/nginx**. | |`--sbin-path=` | Name of the NGINX executable file, which is used only during installation. Default: **/sbin/nginx | |`--conf-path=` | Name of the NGINX configuration file. You can, however, always override this value at startup by specifying a different file with the `-c ` option on the `nginx` command line. Default: **conf/nginx.conf | |`--pid-path=` | Name of the **nginx.pid** file, which stores the process ID of the `nginx` master process. After installation, the path to the filename can be changed with the [pid](https://nginx.org/en/docs/ngx_core_module.html#pid) directive in the NGINX configuration file. Default: **/logs/nginx.pid | |`--error-log-path=` | Name of the primary log file for errors, warnings, and diagnostic data. After installation, the filename can be changed with the [error_log](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive in the NGINX configuration file. Default: **/logs/error.log | |`--http-log-path=` | Name of the primary log file for requests to the HTTP server. After installation, the filename can always be changed with the [access_log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directive in the NGINX configuration file. Default: **/logs/access.log | |`--user=` | Name of the unprivileged user whose credentials are used by the NGINX worker processes. After installation, the name can be changed with the [user](https://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX configuration file. Default: `nobody` | |`--group=` | Name of the group whose credentials are used by the NGINX worker processes. After installation, the name can be changed with the [user](https://nginx.org/en/docs/ngx_core_module.html#user) directive in the NGINX configuration file. Default: the value set by the `--user`` option. | |`--with-pcre=` | Path to the source for the PCRE library, which is required for regular expressions support in the [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive and the [Rewrite](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html) module. | |`--with-pcre-jit` | Builds the PCRE library with “just-in-time compilation” support (the [pcre_jit](https://nginx.org/en/docs/ngx_core_module.html#pcre_jit) directive). | |`--with-zlib=` | Path to the source for the `zlib` library, which is required by the [Gzip](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) module. | ### GCC options With the `configure` script you can also specify compiler‑related options. |Parameter | Description | | ---| --- | |`--with-cc-opt=""` | Additional parameters that are added to the ``CFLAGS`` variable. When using the system PCRE library under FreeBSD, the mandatory value is `--with-cc-opt="-I /usr/local/include"`. If the number of files supported by `select()` needs to be increased, it can also specified here as in this example: `--with-cc-opt="-D FD_SETSIZE=2048"`. | |`--with-ld-opt=""` | Additional parameters that are used during linking. When using the system PCRE library under FreeBSD, the mandatory value is `--with-ld-opt="-L /usr/local/lib"`. | ### Connection processing methods {#configure_methods} With the `configure` script you can redefine the method for event‑based polling. For more information, see [Connection processing methods](https://nginx.org/en/docs/events.html) in the NGINX reference documentation. |Module Name | Description | | ---| --- | |`--with-select_module`, `--without-select_module` | Enables or disables building a module that enable NGINX to work with the ``select()`` method. The modules is built automatically if the platform does not appear to support more appropriate methods such as `kqueue`, `epoll`, or `/dev/poll`. | |`--with-poll_module`, `--without-poll_module` | Enables or disables building a module that enables NGINX to work with the `poll()` method. The module is built automatically if the platform does not appear to support more appropriate methods such as `kqueue`, `epoll`, or `/dev/poll`. | ### Select the NGINX modules to build NGINX consists of a set of function‑specific _modules_, which are specified with `configure` script along with other build options. Some modules are built by default – they do not have to be specified with the `configure` script. Default modules can however be explicitly excluded from the NGINX binary with the`--without-`option on the `configure` script. Modules not included by default, as well as third‑party modules, must be explicitly specified in the `configure` script together with other build options. Such modules can be linked to NGINX binary either _statically_ (they are then loaded each time NGINX starts) or _dynamically_ (they are loaded only if associated directives are included in the NGINX configuration file. #### Modules built by default If you do not need a module that is built by default, you can disable it by naming it with the `--without-` option on the `configure` script, as in this example which disables the [Empty GIF](https://nginx.org/en/docs/http/ngx_http_empty_gif_module.html) module (should be typed as a single line): ```shell ./configure --sbin-path=/usr/local/nginx/nginx \ --conf-path=/usr/local/nginx/nginx.conf \ --pid-path=/usr/local/nginx/nginx.pid \ --with-http_ssl_module \ --with-stream \ --with-pcre=../pcre2-10.43 \ --with-zlib=../zlib-1.3.1 \ --without-http_empty_gif_module ``` |Module Name | Description | | ---| --- | |[`http_access_module`](https://nginx.org/en/docs/http/ngx_http_access_module.html) | Accepts or denies requests from specified client addresses. | |[`http_auth_basic_module`](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) | Limits access to resources by validating the user name and password using the HTTP Basic Authentication protocol. | |[`http_autoindex_module`](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html) | Processes requests ending with the forward-slash character (_/_) and produces a directory listing. | |[`http_browser_module`](https://nginx.org/en/docs/http/ngx_http_browser_module.html) | Creates variables whose values depend on the value of the ``User-Agent`` request header. | |[`http_charset_module`](https://nginx.org/en/docs/http/ngx_http_charset_module.html) | Adds the specified character set to the ``Content-Type`` response header. Can convert data from one character set to another. | |[`http_empty_gif_module`](https://nginx.org/en/docs/http/ngx_http_empty_gif_module.html) | Emits a single-pixel transparent GIF. | |[`http_fastcgi_module`](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html) | Passes requests to a FastCGI server. | |[`http_geo_module`](https://nginx.org/en/docs/http/ngx_http_geo_module.html) | Creates variables with values that depend on the client IP address. | |[`http_gzip_module`](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) | Compresses responses using `gzip`, reducing the amount of transmitted data by half or more. | |[`http_limit_conn_module`](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html) | Limits the number of connections per a defined key, in particular, the number of connections from a single IP address. | |[`http_limit_req_module`](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html) | Limits the request processing rate per a defined key, in particular, the processing rate of requests coming from a single IP address. | |[`http_map_module`](https://nginx.org/en/docs/http/ngx_http_map_module.html) | Creates variables whose values depend on the values of other variables. | |[`http_memcached_module`](https://nginx.org/en/docs/http/ngx_http_memcached_module.html) | Passes requests to a memcached server. | |[`http_proxy_module`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) | Passes HTTP requests to another server. | |[`http_referer_module`](https://nginx.org/en/docs/http/ngx_http_referer_module.html) | Blocks requests with invalid values in the `Referer` header. | |[`http_rewrite_module`](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html) | Changes the request URI using regular expressions and return redirects; conditionally selects configurations. Requires the [PCRE](http://pcre.org/) library. | |[`http_scgi_module`](https://nginx.org/en/docs/http/ngx_http_scgi_module.html) | Passes requests to an SCGI server. | |[`http_ssi_module`](https://nginx.org/en/docs/http/ngx_http_ssi_module.html) | Processes SSI (Server Side Includes) commands in responses passing through it. | |[`http_split_clients_module`](https://nginx.org/en/docs/http/ngx_http_split_clients_module.html) | Creates variables suitable for A/B testing, also known as split testing. | |[`http_upstream_hash_module`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#hash) | Enables the generic Hash load-balancing method. | |[`http_upstream_ip_hash_module`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash) | Enables the IP Hash load-balancing method. | |[`http_upstream_keepalive_module`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) | Enables keepalive connections. | |[`http_upstream_least_conn_module`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_conn) | Enables the Least Connections load-balancing method. | |[`http_upstream_zone_module`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) | Enables shared memory zones. | |[`http_userid_module`](https://nginx.org/en/docs/http/ngx_http_userid_module.html) | Sets cookies suitable for client identification. | |[`http_uwsgi_module`](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html) | Passes requests to a uwsgi server. | ### Include modules not built by default Many NGINX modules are not built by default, and must be listed on the `configure` command line to be built. The [mail](https://nginx.org/en/docs/mail/ngx_mail_core_module.html), [stream](https://nginx.org/en/docs/stream/ngx_stream_core_module.html), [geoip](https://nginx.org/en/docs/http/ngx_http_geoip_module.html), [image_filter](https://nginx.org/en/docs/http/ngx_http_image_filter_module.html), [perl](https://nginx.org/en/docs/http/ngx_http_perl_module.html) and [xslt](https://nginx.org/en/docs/http/ngx_http_xslt_module.html) modules can be compiled as dynamic. See [Dynamic Modules](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) for details. An example of the `configure` command that includes nondefault modules (should be typed as a single line): ```shell ./configure \ --sbin-path=/usr/local/nginx/nginx \ --conf-path=/usr/local/nginx/nginx.conf \ --pid-path=/usr/local/nginx/nginx.pid \ --with-pcre=../pcre2-10.43 \ --with-zlib=../zlib-1.3.1 \ --with-http_ssl_module \ --with-stream \ --with-mail ``` } |Module Name | Description | | -------------------------| ---------------------------------------------| |`--with-cpp_test_module` | Tests the C++ compatibility of header files.| |`--with-debug` | Enables the [debugging log](/nginx/admin-guide/monitoring/debugging.md)| | `--with-file-aio` |Enables asynchronous I/O. | | `--with-google-perftools` | Allows using [Google Performance tools](https://github.com/gperftools/gperftools) library. | |[`--with-http_addition_module`](https://nginx.org/en/docs/http/ngx_http_addition_module.html)| Adds text before and after a response. | |[`--with-http_auth_request_module`](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html)|Implements client authorization based on the result of a subrequest. | |[`--with-http_dav_module`](https://nginx.org/en/docs/http/ngx_http_dav_module.html)|Enables file management automation using the WebDAV protocol. | |`--with-http_degradation_module`|Allows returning an error when a memory size exceeds the defined value. | |[`--with-http_flv_module`](https://nginx.org/en/docs/http/ngx_http_flv_module.html)|Provides pseudo-streaming server-side support for Flash Video (FLV) files. | |[`--with-http_geoip_module`](https://nginx.org/en/docs/http/ngx_http_geoip_module.html)|Enables creating variables whose values depend on the client IP address. The module uses [MaxMind](http://www.maxmind.com) GeoIP databases. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-http_geoip_module=dynamic`. | |[`--with-http_gunzip_module`](https://nginx.org/en/docs/http/ngx_http_gunzip_module.html)|Decompresses responses with `Content-Encoding: gzip` for clients that do not support the _zip_ encoding method. | |[`--with-http_gzip_static_module`](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html)| Allows sending precompressed files with the **.gz** filename extension instead of regular files. | |[`--with-http_image_filter_module`](https://nginx.org/en/docs/http/ngx_http_image_filter_module.html)|T ransforms images in JPEG, GIF, and PNG formats. The module requires the [LibGD](http://libgd.github.io/) library. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-http_image_filter_module=dynamic`. | |[`--with-http_mp4_module`](https://nginx.org/en/docs/http/ngx_http_mp4_module.html)| Provides pseudo-streaming server-side support for MP4 files. | |[`--with-http_perl_module`](https://nginx.org/en/docs/http/ngx_http_perl_module.html)| Used to implement location and variable handlers in Perl and insert Perl calls into SSI. Requires the [PERL](https://www.perl.org/get.html) library. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-http_perl_module=dynamic`. | |[`--with-http_random_index_module`](https://nginx.org/en/docs/http/ngx_http_random_index_module.html) | Processes requests ending with the slash character (‘/’) and picks a random file in a directory to serve as an index file. | |[`--with-http_realip_module`](https://nginx.org/en/docs/http/ngx_http_realip_module.html) | Changes the client address to the one sent in the specified header field. | |[`--with-http_secure_link_module`](https://nginx.org/en/docs/http/ngx_http_secure_link_module.html) | Used to check authenticity of requested links, protect resources from unauthorized access, and limit link lifetime. | |[`--with-http_slice_module`](https://nginx.org/en/docs/http/ngx_http_slice_module.html) | Allows splitting a request into subrequests, each subrequest returns a certain range of response. Provides more effective caching of large files. | |[`--with-http_ssl_module`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) | Enables HTTPS support. Requires an SSL library such as [OpenSSL](https://www.openssl.org/). | |[`--with-http_stub_status_module`](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html)| Provides access to basic status information. Note that NGINX Plus customers do not require this module as they are already provided with extended status metrics and interactive dashboard. | |[`--with-http_sub_module`](https://nginx.org/en/docs/http/ngx_http_sub_module.html) | Modifies a response by replacing one specified string by another. | |[`--with-http_xslt_module`](https://nginx.org/en/docs/http/ngx_http_xslt_module.html)| Transforms XML responses using one or more XSLT stylesheets. The module requires the [Libxml2](http://xmlsoft.org/) and [XSLT](http://xmlsoft.org/XSLT/) libraries. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-http_xslt_module=dynamic`. | |[`--with-http_v2_module`](https://nginx.org/en/docs/http/ngx_http_v2_module.html)| Enable support for [HTTP/2](https://datatracker.ietf.org/doc/html/rfc7540). See [The HTTP/2 Module in NGINX](https://www.nginx.com/blog/http2-module-nginx/) on the NGINX blog for details. | | [`--with-mail`](https://nginx.org/en/docs/mail/ngx_mail_core_module.html)| Enables mail proxy functionality. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-mail=dynamic`. | |[`--with-mail_ssl_module`](https://nginx.org/en/docs/mail/ngx_mail_ssl_module.html)| Provides support for a mail proxy server to work with the SSL/TLS protocol. Requires an SSL library such as [OpenSSL](https://www.openssl.org/). | | [`--with-stream`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html) | Enables the TCP and UDP proxy functionality. To compile as a separate [dynamic module](/nginx/admin-guide/dynamic-modules/dynamic-modules.md) instead, change the option to `--with-stream=dynamic`. | | [`--with-stream_ssl_module`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html)| Provides support for a stream proxy server to work with the SSL/TLS protocol. Requires an SSL library such as [OpenSSL](https://www.openssl.org/). | | `--with-threads` | Enables NGINX to use thread pools. For details, see [Thread Pools in NGINX Boost Performance 9x!](https://www.nginx.com/blog/thread-pools-boost-performance-9x/) on the NGINX blog. | ### Include third-party modules You can extend NGINX functionality by compiling NGINX Open Source with your own module or a third‑party module. Some third‑party modules are listed in the [NGINX Wiki](https://nginx.com/resources/wiki/modules/). Use third‑party modules at your own risk as their stability is not guaranteed. #### Statically linked modules Most modules built into NGINX Open Source are _statically linked_: they are built into NGINX Open Source at compile time and are linked to the NGINX binary statically. These modules can be disabled only by recompiling NGINX. To compile NGINX Open Source with a statically linked third‑party module, include the `--add-module=` option on the `configure` command, where `` is the path to the source code (this example is for the [RTMP](https://github.com/arut/nginx-rtmp-module) module): ```shell ./configure ... --add-module=/usr/build/nginx-rtmp-module ``` #### Dynamically linked modules NGINX modules can also be compiled as a shared object (**\*.so** file) and then dynamically loaded into NGINX Open Source at runtime. This provides more flexibility, as the module can be loaded or unloaded at any time by adding or removing the associated [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive in the NGINX configuration file and reloading the configuration. Note that the module itself must support dynamic linking. To compile NGINX Open Source with a dynamically loaded third‑party module, include the `--add-dynamic-module=` option on the `configure` command, where `` is the path to the source code: ```shell ./configure ... --add-dynamic-module= ``` The resulting **\*.so** files are written to the _prefix_**/modules/** directory, where the _prefix_ is a directory for server files such as **/usr/local/nginx/**. To load a dynamic module, add the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive to the NGINX configuration after installation: ```nginx load_module modules/ngx_mail_module.so; ``` For more information, see [Compiling Third‑Party Dynamic Modules for NGINX and NGINX Plus](https://www.nginx.com/blog/compiling-dynamic-modules-nginx-plus/) on the NGINX blog and [Extending NGINX](https://nginx.com/resources/wiki/extending/) in the Wiki. ### Complete the installation from source - Compile and install the build: ```shell make sudo make install ``` - After the installation is finished, start NGINX Open Source: ```shell sudo nginx ``` -------------------------------------------------------------------------------- title: "Serve Static Content" url: https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/index.md description: Configure NGINX and F5 NGINX Plus to serve static content, with type-specific root directories, checks for file existence, and performance optimizations. -------------------------------------------------------------------------------- This section describes how to configure NGINX and F5 NGINX Plus to serve static content, how to define which paths are searched to find requested files, how to set up index files, and how to tune NGINX and NGINX Plus, as well as the kernel, for optimal performance. ## Root Directory and Index Files The [root](https://nginx.org/en/docs/http/ngx_http_core_module.html#root) directive specifies the root directory that will be used to search for a file. To obtain the path of a requested file, NGINX appends the request URI to the path specified by the `root` directive. The directive can be placed on any level within the `http {}`, `server {}`, or `location {}` contexts. In the example below, the `root` directive is defined for a virtual server. It applies to all `location {}` blocks where the `root` directive is not included to explicitly redefine the root: ```nginx server { root /www/data; location / { } location /images/ { } location ~ \.(mp3|mp4) { root /www/media; } } ``` Here, NGINX searches for a URI that starts with `/images/` in the `/www/data/images/` directory in the file system. But if the URI ends with the `.mp3` or `.mp4` extension, NGINX instead searches for the file in the `/www/media/` directory because it is defined in the matching `location` block. If a request ends with a slash, NGINX treats it as a request for a directory and tries to find an index file in the directory. The [index](https://nginx.org/en/docs/http/ngx_http_index_module.html#index) directive defines the index file’s name (the default value is `index.html`). To continue with the example, if the request URI is `/images/some/path/`, NGINX delivers the file `/www/data/images/some/path/index.html` if it exists. If it does not, NGINX returns HTTP code `404 (Not Found)` by default. To configure NGINX to return an automatically generated directory listing instead, include the `on` parameter to the [autoindex](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html#autoindex) directive: ```nginx location /images/ { autoindex on; } ``` You can list more than one filename in the `index` directive. NGINX searches for files in the specified order and returns the first one it finds. ```nginx location / { index index.$geo.html index.htm index.html; } ``` The `$geo` variable used here is a custom variable set through the [geo](https://nginx.org/en/docs/http/ngx_http_geo_module.html#geo) directive. The value of the variable depends on the client’s IP address. To return the index file, NGINX checks for its existence and then makes an internal redirect to the URI obtained by appending the name of the index file to the base URI. The internal redirect results in a new search of a location and can end up in another location as in the following example: ```nginx location / { root /data; index index.html index.php; } location ~ \.php { fastcgi_pass localhost:8000; #... } ``` Here, if the URI in a request is `/path/`, and `/data/path/index.html` does not exist but `/data/path/index.php` does, the internal redirect to `/path/index.php` is mapped to the second location. As a result, the request is proxied. ## Trying Several Options The [try_files](https://nginx.org/en/docs/http/ngx_http_core_module.html#try_files) directive can be used to check whether the specified file or directory exists; NGINX makes an internal redirect if it does, or returns a specified status code if it doesn’t. For example, to check the existence of a file corresponding to the request URI, use the `try_files` directive and the `$uri` variable as follows: ```nginx server { root /www/data; location /images/ { try_files $uri /images/default.gif; } } ``` The file is specified in the form of the URI, which is processed using the `root` or `alias` directives set in the context of the current location or virtual server. In this case, if the file corresponding to the original URI doesn’t exist, NGINX makes an internal redirect to the URI specified by the last parameter, returning `/www/data/images/default.gif`. The last parameter can also be a status code (directly preceded by the equals sign) or the name of a location. In the following example, a `404` error is returned if none of the parameters to the `try_files` directive resolve to an existing file or directory. ```nginx location / { try_files $uri $uri/ $uri.html =404; } ``` In the next example, if neither the original URI nor the URI with the appended trailing slash resolve into an existing file or directory, the request is redirected to the named location which passes it to a proxied server. ```nginx location / { try_files $uri $uri/ @backend; } location @backend { proxy_pass http://backend.example.com; } ``` For more information, watch the [Content Caching](https://www.nginx.com/resources/webinars/content-caching-nginx-plus/) webinar on‑demand to learn how to dramatically improve the performance of a website, and get a deep‑dive into NGINX’s caching capabilities. ## Optimize Performance for Serving Content Loading speed is a crucial factor of serving any content. Making minor optimizations to your NGINX configuration may boost the productivity and help reach optimal performance. ### Enable `sendfile` By default, NGINX handles file transmission itself and copies the file into the buffer before sending it. Enabling the [sendfile](https://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile) directive eliminates the step of copying the data into the buffer and enables direct copying data from one file descriptor to another. Alternatively, to prevent one fast connection from entirely occupying the worker process, you can use the [sendfile_max_chunk](https://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile_max_chunk) directive to limit the amount of data transferred in a single `sendfile()` call (in this example, to `1` MB): ```nginx location /mp3 { sendfile on; sendfile_max_chunk 1m; #... } ``` ### Enable `tcp_nopush` Use the [tcp_nopush](https://nginx.org/en/docs/http/ngx_http_core_module.html#tcp_nopush) directive together with the [sendfile](https://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile) `on;`directive. This enables NGINX to send HTTP response headers in one packet right after the chunk of data has been obtained by `sendfile()`. ```nginx location /mp3 { sendfile on; tcp_nopush on; #... } ``` ### Enable `tcp_nodelay` The [tcp_nodelay](https://nginx.org/en/docs/http/ngx_http_core_module.html#tcp_nodelay) directive allows override of [Nagle’s algorithm](https://en.wikipedia.org/wiki/Nagle's_algorithm), originally designed to solve problems with small packets in slow networks. The algorithm consolidates a number of small packets into a larger one and sends the packet with a `200` ms delay. Nowadays, when serving large static files, the data can be sent immediately regardless of the packet size. The delay also affects online applications (ssh, online games, online trading, and so on). By default, the [tcp_nodelay](https://nginx.org/en/docs/http/ngx_http_core_module.html#tcp_nodelay) directive is set to `on` which means that the Nagle’s algorithm is disabled. Use this directive only for keepalive connections: ```nginx location /mp3 { tcp_nodelay on; keepalive_timeout 65; #... } ``` ### Optimize the Backlog Queue One of the important factors is how fast NGINX can handle incoming connections. The general rule is when a connection is established, it is put into the “listen” queue of a listen socket. Under normal load, either the queue is small or there is no queue at all. But under high load, the queue can grow dramatically, resulting in uneven performance, dropped connections, and increased latency. #### Display the Listen Queue To display the current listen queue, run this command: ```none netstat -Lan ``` The output might be like the following, which shows that in the listen queue on port `80` there are `10` unaccepted connections against the configured maximum of `128` queued connections. This situation is normal. ```none Current listen queue sizes (qlen/incqlen/maxqlen) Listen Local Address 0/0/128 *.12345 10/0/128 *.80 0/0/128 *.8080 ``` In contrast, in the following command the number of unaccepted connections (`192`) exceeds the limit of `128`. This is quite common when a web site experiences heavy traffic. To achieve optimal performance, you need to increase the maximum number of connections that can be queued for acceptance by NGINX in both your operating system and the NGINX configuration. ```none Current listen queue sizes (qlen/incqlen/maxqlen) Listen Local Address 0/0/128 *.12345 192/0/128 *.80 0/0/128 *.8080 ``` #### Tune the Operating System Increase the value of the `net.core.somaxconn` kernel parameter from its default value (`128`) to a value high enough for a large burst of traffic. In this example, it's increased to `4096`. - For FreeBSD, run the command: ```none sudo sysctl kern.ipc.somaxconn=4096 ``` - For Linux: 1. Run the command: ```none sudo sysctl -w net.core.somaxconn=4096 ``` 2. Use a text editor to add the following line to `/etc/sysctl.conf`: ```none net.core.somaxconn = 4096 ``` #### Tune NGINX If you set the `somaxconn` kernel parameter to a value greater than `512`, change the `backlog` parameter to the NGINX [listen](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) directive to match: ```nginx server { listen 80 backlog=4096; # ... } ``` -------------------------------------------------------------------------------- title: "Single Sign-On with Amazon Cognito" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/cognito/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Amazon Cognito as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Amazon Cognito](https://aws.amazon.com/cognito/) as the Identity Provider (IdP), and NGINX Plus as the Relying Party, or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. In the latest NGINX Plus version, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - An [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) - A Cognito **User Pool** - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure Amazon Cognito {#cognito-setup} 1. Open the Amazon Cognito console in the AWS Management Console 2. In the Cognito dashboard, select **Create or open a User Pool**. 3. **Create an App Client** (for example, "nginx-demo-app") and **enable the "Generate client secret"** option. 4. In the App client settings, select **Enable Cognito User Pool** as an **Identity Provider**: - Add a **Callback URL**: `https://demo.example.com/oidc_callback`. - Enable **Authorization code grant**. - In the **OAuth scopes**, check the values of **openid**, **profile**, and **email**. 5. Configure logout URLs to support RP-initiated logout: - Add a **Sign out URL**: `https://demo.example.com/post_logout/`. 6. Copy the following values — you will need them later when configuring NGINX Plus. - **User Pool ID**, for example, `us-east-2_abCdEfGhI` - **App client id** and **App client secret** - **AWS region**, for example, `us-east-2` - **Issuer**, for example, `https://cognito-idp..amazonaws.com/` ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, Amazon Cognito publishes the `.well-known/openid-configuration` document at the following address: `https://cognito-idp..amazonaws.com//.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI/.well-known/openid-configuration | jq ``` Where: - the `cognito-idp.us-east-2.amazonaws.com` is your Amazon Cognito server address - the `us-east-2_abCdEfGhI` is your User Pool ID - the `/.well-known/openid-configuration` is the default address for Amazon Cognito for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI", "authorization_endpoint": "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI/oauth2/authorize", "token_endpoint": "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI/oauth2/token", "jwks_uri": "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI/.well-known/jwks.json", "userinfo_endpoint": "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI/oauth2/userInfo", "end_session_endpoint": "https://your-cognito-domain.auth.us-east-2.amazoncognito.com/logout", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for Amazon Cognito is `https://cognito-idp..amazonaws.com/`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ## Set up NGINX Plus {#nginx-plus-setup} With Cognito configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [Cognito Configuration](#cognito-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the Amazon Cognito provider named `cognito` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider cognito { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - your actual Amazon Cognito **Client ID** obtained in [Amazon Cognito Configuration](#cognito-setup) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - your **Client Secret** obtained in [Amazon Cognito Configuration](#cognito-setup) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - the **Issuer** URL obtained in [Amazon Cognito Configuration](#cognito-setup) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your Amazon Cognito OIDC URL. As a rule, Cognito uses a unique issuer for each User Pool, for example: `https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI` By default, NGINX Plus creates the metadata URL by appending the `/.well-known/openid-configuration` part to the Issuer URL. If your Issuer is different, you can explicitly specify the metadata document with the [`config_url`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url) directive. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where Amazon Cognito should redirect the user after a successful logout. This value **must also be configured** in the Cognito App client sign out URLs. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch `/oauth2/userInfo` from the Amazon Cognito and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when Cognito's OpenID Connect discovery document advertises the `S256` code challenge method in the `code_challenge_methods_supported` field. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP metadata does not list `S256`. - The module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`). - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider cognito { issuer https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with Amazon Cognito OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `cognito` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#cognito-setup-oidc-provider): ```nginx # ... location / { auth_oidc cognito; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by Amazon Cognito: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by Amazon Cognito - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc cognito; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nCognito sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider cognito { # Typically your Cognito issuer is something like: # https://cognito-idp..amazonaws.com/ issuer https://cognito-idp.us-east-2.amazonaws.com/us-east-2_abCdEfGhI; # Your Cognito "App client id" and "App client secret" client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Protect this path with Cognito OIDC auth_oidc cognito; # Forward OIDC claims as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nCognito sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open https://demo.example.com/ in a browser. You will be automatically redirected to Amazon Cognito login page for your realm. 2. Enter valid Cognito user credentials (those in the assigned user pool). Upon successful sign-in, Cognito redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; Amazon Cognito ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to Amazon Cognito for a fresh sign‑in, proving the session has been terminated. ## Legacy njs-based Amazon Cognito Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based Cognito Guide](nginx/deployment-guides/single-sign-on/oidc-njs/cognito.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added PKCE configuration (`pkce` directive) and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "SSL Termination for TCP Upstream Servers" url: https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/index.md description: Terminate SSL/TLS-encrypted traffic from clients, relieving your upstream TCP servers of the computational load. -------------------------------------------------------------------------------- This article explains how to set up SSL termination for F5 NGINX Plus and a load-balanced group of servers that accept TCP connections. ## What is SSL Termination? SSL termination means that NGINX Plus acts as the server-side SSL endpoint for connections with clients: it performs the decryption of requests and encryption of responses that backend servers would otherwise have to do. The operation is called termination because NGINX Plus closes the client connection and forwards the client data over a newly created, unencrypted connection to the servers in an upstream group. In release R6 and later, NGINX Plus performs SSL termination for TCP connections as well as HTTP connections. ## Prerequisites - [NGINX Plus R6](nginx/releases.md#r6 ) or later - A load-balanced [upstream group](nginx/admin-guide/load-balancer/tcp-udp-load-balancer.md) with several TCP servers - SSL certificates and a private key (obtained or self-generated) ## Obtaining SSL Certificates First, you will need to obtain server certificates and a private key and put them on the server. A certificate can be obtained from a trusted certificate authority (CA) or generated using an SSL library such as [OpenSSL](https://www.openssl.org/). ## Configuring NGINX Plus To configure SSL termination, add the following directives to the NGINX Plus configuration: ### Enabling SSL To enable SSL, specify the `ssl` parameter of the [listen](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#listen) directive for the TCP server that passes connections to an upstream server group: ```nginx stream { server { listen 12345 ssl; proxy_pass backend; #... } } ``` ### Adding SSL Certificates To add SSL certificates, specify the path to the certificates (which must be in the PEM format) with the [ssl_certificate](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_certificate) directive, and specify the path to the private key in the [ssl_certificate_key](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_certificate_key) directive: ```nginx server { #... ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; } ``` Additionally, the [ssl_protocols](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_protocols) and [ssl_ciphers](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_ciphers) directives can be used to limit connections and to include only the strong versions and ciphers of SSL/TLS: ```nginx server { #... ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; } ``` The [ssl_ciphers](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_ciphers) directive tells NGINX to inform the SSL library which ciphers it prefers. ## Speeding up Secure TCP Connections Implementing SSL/TLS can significantly impact server performance, because the SSL handshake operation (a series of messages the client and server exchange to verify that the connection is trusted) is quite CPU-intensive. The default timeout for the SSL handshake is 60 seconds and it can be redefined with the [ssl_handshake_timeout](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_handshake_timeout) directive. We do not recommend setting this value too low or too high, as that might result either in handshake failure or a long time to wait for the handshake to complete: ```nginx server { #... ssl_handshake_timeout 10s; } ``` ### Optimizing the SSL Session Cache Creating a cache of the session parameters that apply to each SSL/TLS connection reduces the number of handshakes and thus can significantly improve performance. Caching is set with the [ssl_session_cache](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_session_cache) directive: ```nginx ssl_session_cache; ``` By default, NGINX Plus uses the `built-in` type of the session cache, which means the cache built in your SSL library. This is not optimal, because such a cache can be used by only one worker process and can cause memory fragmentation. Set the `ssl_session_cache` directive to `shared` to share the cache among all worker processes, which speeds up later connections because the connection setup information is already known: ```nginx ssl_session_cache shared:SSL:1m; ``` As a reference, a 1-MB shared cache can hold approximately 4,000 sessions. By default, NGINX Plus retains cached session parameters for five minutes. Increasing the value of the [ssl_session_timeout](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_session_timeout) to several hours can improve performance because reusing cached session parameters reduces the number of time-consuming handshakes. When you increase the timeout, the cache needs to be bigger to accommodate the larger number of cached parameters that results. For the 4-hour timeout in the following example, a 20-MB cache is appropriate: ```nginx ssl_session_timeout 4h; ``` If the timeout length is increased, you need a larger cache to store sessions, for example, 20 MB: ```nginx server { #... ssl_session_cache shared:SSL:20m; ssl_session_timeout 4h; } ``` These lines create an in-memory cache of 20 MB to store session information, and instruct NGINX Plus to reuse session parameters from the cache for 4 hours after the moment they were added. ### Session Tickets Session tickets are an alternative to the session cache. Session information is stored on the client side, eliminating the need for a server-side cache to store session information. When a client resumes interaction with the backend server, it presents the session ticket and re-negotiation is not necessary. Set the [ssl_session_tickets](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_session_tickets) directive to `on`: ```nginx server { #... ssl_session_tickets on; } ``` When using session tickets for an upstream group, each upstream server must be initialized with the same session key. It’s a best practice to change session keys frequently, we recommend that you implement a mechanism to rotate the shared key across all upstream servers: ```nginx server { #... ssl_session_tickets on; ssl_session_ticket_key /etc/ssl/session_ticket_keys/current.key; ssl_session_ticket_key /etc/ssl/session_ticket_keys/previous.key; } ``` ## Complete Example ```nginx stream { upstream stream_backend { server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12345; } server { listen 12345 ssl; proxy_pass stream_backend; ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; ssl_session_cache shared:SSL:20m; ssl_session_timeout 4h; ssl_handshake_timeout 30s; #... } } ``` In this example, the directives in the `server` block instruct NGINX Plus to terminate and decrypt secured TCP traffic from clients and pass it unencrypted to the upstream group `stream_backend` which consists of three servers. The `ssl` parameter of the [listen](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#listen) directive instructs NGINX Plus to accept SSL connections. When a clent requests a secure TCP connection, NGINX Plus starts the handshake process, which uses the PEM-format certificate specified by the [ssl_certificate](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_certificate) directive, the certificate’s private key specified by the [ssl_certificate_key](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_certificate_key) directive, and the protocols and cyphers listed by the [ssl_protocols](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_protocols) and [ssl_ciphers](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_ciphers) directives. As soon as the secure TCP connection is established, NGINX Plus caches the session parameters according to the [ssl_session_cache](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_session_cache) directive. In the example, the session cache is shared between all worker processes (the `shared` parameter), is 20 MB in size (the `20m` parameter), and retains each SSL session for reuse for 4 hours (the [ssl_session_timeout](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_session_timeout) directive). -------------------------------------------------------------------------------- title: "TCP and UDP Load Balancing" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/index.md description: This chapter describes how to use F5 NGINX Plus and NGINX Open Source to proxy and load balance TCP and UDP traffic. -------------------------------------------------------------------------------- ## Introduction {#intro} [Load balancing](https://www.f5.com/glossary/load-balancer) is a traffic management technique that efficiently distributes network traffic across multiple backend servers. NGINX Plus and NGINX Open Source provide Layer 4 (transport layer) proxying and load balancing for both Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) traffic. TCP is a connection-oriented protocol used by services such as LDAP, MySQL, and RTMP. UDP is commonly used by non-transactional, latency-sensitive services such as DNS, syslog, and RADIUS. For Layer 7, or application-layer HTTP load balancing, see [HTTP Load Balancing](http-load-balancer.md). ## Prerequisites - NGINX Plus (no extra build steps required) or [NGINX Open Source](https://nginx.org/en/download.html) (either built with the [`--with-stream`](https://nginx.org/en/docs/configure.html) configuration flag or loaded as a dynamic module). - An application, database, or service that communicates over TCP or UDP - Upstream servers, each running the same instance of the application, database, or service ## Configuring reverse proxy {#proxy_pass} First, you will need to configure _reverse proxy_ so that NGINX Plus or NGINX Open Source can forward TCP connections or UDP datagrams from clients to an upstream group or a proxied server. Open the NGINX configuration file and perform the following steps: 1. Create a top‑level [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) block: ```nginx stream { # ... } ``` 2. Define one or more [`server {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#server) configuration blocks for each virtual server in the top‑level `stream {}` context. 3. Within the `server {}` configuration block for each server, include the [`listen`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#listen) directive to define the _IP address_ and/or _port_ on which the server listens. For UDP traffic, also include the [`udp`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#udp) parameter. As TCP is the default protocol for the `stream` context, there is no `tcp` parameter to the `listen` directive: ```nginx stream { server { listen 12345; # ... } server { listen 53 udp; # ... } # ... } ``` 4. Include the [`proxy_pass`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_pass) directive to define the proxied server or an upstream group to which the server forwards traffic: ```nginx stream { server { listen 12345; #TCP traffic will be forwarded to the "stream_backend" upstream group proxy_pass stream_backend; } server { listen 12346; #TCP traffic will be forwarded to the specified server proxy_pass backend.example.com:12346; } server { listen 53 udp; #UDP traffic will be forwarded to the "dns_servers" upstream group proxy_pass dns_servers; } # ... } ``` 5. If the proxy server has several network interfaces, you can optionally configure NGINX to use a particular source IP address when connecting to an upstream server. This may be useful if a proxied server behind NGINX is configured to accept connections from particular IP networks or IP address ranges. Include the [`proxy_bind`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_bind) directive and the IP address of the appropriate network interface: ```nginx stream { # ... server { listen 127.0.0.1:12345; proxy_pass backend.example.com:12345; proxy_bind 127.0.0.1:12345; } } ``` 6. Optionally, you can tune the size of two in‑memory buffers where NGINX can put data from both the client and upstream connections. If there is a small volume of data, the buffers can be reduced which may save memory resources. If there is a large volume of data, the buffer size can be increased to reduce the number of socket read/write operations. As soon as data is received on one connection, NGINX reads it and forwards it over the other connection. The buffers are controlled with the [`proxy_buffer_size`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_buffer_size) directive: ```nginx stream { # ... server { listen 127.0.0.1:12345; proxy_pass backend.example.com:12345; proxy_buffer_size 16k; } } ``` ## Configuring TCP or UDP load balancing {#upstream} To configure load balancing: 1. Create a group of servers, or an _upstream group_ whose traffic will be load balanced. Define one or more [`upstream {}`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) configuration blocks in the top‑level [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context and set the name for the upstream group, for example, `stream_backend` for TCP servers and `dns_servers` for UDP servers: ```nginx stream { upstream stream_backend { # ... } upstream dns_servers { # ... } # ... } ``` Make sure that the name of the upstream group is referenced by a [`proxy_pass`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_pass) directive, like those configured [above](#proxy_pass) for reverse proxy. 2. Populate the upstream group with _upstream servers_. Within the [`upstream {}`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) block, add a [`server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) directive for each upstream server, specifying its IP address or hostname (which can resolve to multiple IP addresses) and an _obligatory_ port number. Note that you do not define the protocol for each server, because that is defined for the entire upstream group by the parameter you include on the `listen` directive in the `server` block, which you have created [earlier](#proxy_pass). ```nginx stream { upstream stream_backend { server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12346; # ... } upstream dns_servers { server 192.168.136.130:53; server 192.168.136.131:53; # ... } # ... } ``` 3. Configure the load‑balancing method used by the upstream group. You can specify one of the following methods: - Round Robin – By default, NGINX uses the Round Robin algorithm to load balance traffic, directing it sequentially to the servers in the configured upstream group. Because it is the default method, there is no `round‑robin` directive; simply create an `upstream {}` configuration block in the top‑level `stream {}` context and add `server` directives as described in the previous step. - [Least Connections](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#least_conn) – NGINX selects the server with the smaller number of current active connections. - [Least Time](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#least_time) – NGINX selects the server with the lowest average latency and the least number of active connections. The method used to calculate lowest average latency depends on which of the following parameters is included on the `least_time` directive: - `connect` – Time to connect to the upstream server - `first_byte` – Time to receive the first byte of data - `last_byte` – Time to receive the full response from the server ```nginx upstream stream_backend { least_time first_byte; server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12346; } ``` - [Hash](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#hash) – NGINX selects the server based on a user‑defined key, for example, the source IP address ([`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr)): ```nginx upstream stream_backend { hash $remote_addr; server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12346; } ``` The `Hash` load‑balancing method is also used to configure _session persistence_. As the hash function is based on client IP address, connections from a given client are always passed to the same server unless the server is down or otherwise unavailable. Specify an optional `consistent` parameter to apply the [ketama](http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients) consistent hashing method: ```nginx hash $remote_addr consistent; ``` - [Random](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#random) – Each connection will be passed to a randomly selected server. If the `two` parameter is specified, first, NGINX randomly selects two servers taking into account server weights, and then chooses one of these servers using the specified method: - `least_conn` – The least number of active connections - `least_time=connect` (NGINX Plus) – The time to connect to the upstream server ([`$upstream_connect_time`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#var_upstream_connect_time)) - `least_time=first_byte` (NGINX Plus) – The least average time to receive the first byte of data from the server ([`$upstream_first_byte_time`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#var_upstream_first_byte_time)) - `least_time=last_byte` (NGINX Plus) – The least average time to receive the last byte of data from the server ([`$upstream_session_time`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#var_upstream_session_time)) ```nginx upstream stream_backend { random two least_time=last_byte; server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12346; server backend4.example.com:12346; } ``` The **Random** load balancing method should be used for distributed environments where multiple load balancers are passing requests to the same set of backends. For environments where the load balancer has a full view of all requests, use other load balancing methods, such as round robin, least connections and least time. 5. Optionally, for each upstream server specify server‑specific parameters including [maximum number of connections](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_conns), [server weight](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#weight), and so on: ```nginx upstream stream_backend { hash $remote_addr consistent; server backend1.example.com:12345 weight=5; server backend2.example.com:12345; server backend3.example.com:12346 max_conns=3; } upstream dns_servers { least_conn; server 192.168.136.130:53; server 192.168.136.131:53; # ... } ``` An alternative approach is to proxy traffic to a single server instead of an upstream group. If you identify the server by hostname, and configure the hostname to resolve to multiple IP addresses, then NGINX load balances traffic across the IP addresses using the `Round Robin` algorithm. In this case, you _must_ specify the server’s port number in the `proxy_pass` directive and _must not_ specify the protocol before IP address or hostname: ```nginx stream { # ... server { listen 12345; proxy_pass backend.example.com:12345; } } ``` ## Configuring health checks {#health} NGINX can continually test your TCP or UDP upstream servers, avoid the servers that have failed, and gracefully add the recovered servers into the load‑balanced group. See [TCP Health Checks](nginx/admin-guide/load-balancer/tcp-health-check.md) for instructions how to configure health checks for TCP. See [UDP Health Checks](nginx/admin-guide/load-balancer/udp-health-check.md) for instructions how to configure health checks for UDP. ## On-the-fly configuration Upstream server groups can be easily reconfigured on-the-fly using NGINX Plus REST API. Using this interface, you can view all servers in an upstream group or a particular server, modify server parameters, and add or remove upstream servers. To enable on-the-fly configuration: 1. Create the top-level `http {}` block or make sure it is present in your configuration: ```nginx http { # ... } ``` 2. Create a location for configuration requests, for example, _api_: ```nginx http { server { location /api { # ... } } } ``` 3. In this location specify the [`api`](https://nginx.org/en/docs/http/ngx_http_api_module.html#api) directive: ```nginx http { server { location /api { api; # ... } } } ``` 4. By default, the NGINX Plus API provides read-only access to data. The `write=on` parameter enables read/write access so that changes can be made to upstreams: ```nginx http { server { location /api { api write=on; # ... } } } ``` 5. Limit access to this location with [`allow`](https://nginx.org/en/docs/http/ngx_http_access_module.html#allow) and [`deny`](https://nginx.org/en/docs/http/ngx_http_access_module.html#deny) directives: ```nginx http { server { location /api { api write=on; allow 127.0.0.1; # permit access from localhost deny all; # deny access from everywhere else } } } ``` 6. When the API is enabled in the write mode, it is recommended restricting access to `PATCH`, `POST`, and `DELETE` methods to particular users. This can be done by implementing [HTTP basic authentication](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html): ```nginx http { server { location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } api write=on; allow 127.0.0.1; deny all; } } } ``` 7. Create a _shared memory zone_ for the group of upstream servers so that all worker processes can use the same configuration. To do this, in the top-level [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) block, find the target upsteam group, add the [`zone`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#zone) directive to the upstream server group and specify the zone name (here, **stream_backend**) and the amount of memory (64 KB): ```nginx stream { upstream stream_backend { zone backend 64k; # ... } } ``` ### On-the-fly configuration example ```nginx stream { # ... # Configuration of an upstream server group upstream appservers { zone appservers 64k; server appserv1.example.com:12345 weight=5; server appserv2.example.com:12345 fail_timeout=5s; server backup1.example.com:12345 backup; server backup2.example.com:12345 backup; } server { # Server that proxies connections to the upstream group proxy_pass appservers; health_check; } } http { # ... server { # Location for API requests location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } api write=on; allow 127.0.0.1; deny all; } } } ``` Here, access to the location is allowed only from the localhost address (`127.0.0.1`). Access from all other IP addresses is denied. To pass a configuration command to NGINX, send an API command by any method, for example, with curl. For example, to add a new server to the server group, send a `POST` request: ```shell curl -X POST -d '{ \ "server": "appserv3.example.com:12345", \ "weight": 4 \ }' -s 'http://127.0.0.1/api/9/stream/upstreams/appservers/servers' ``` To remove a server from the server group, send a `DELETE` request: ```shell curl -X DELETE -s 'http://127.0.0.1/api/9/stream/upstreams/appservers/servers/0' ``` To modify a parameter for a specific server, send a `PATCH` request: ```shell curl -X PATCH -d '{ "down": true }' -s 'http://127.0.0.1/api/9/http/upstreams/appservers/servers/0' ``` ## Example of TCP and UDP load-balancing configuration {#example} This is a configuration example of TCP and UDP load balancing with NGINX: ```nginx stream { upstream stream_backend { least_conn; server backend1.example.com:12345 weight=5; server backend2.example.com:12345 max_fails=2 fail_timeout=30s; server backend3.example.com:12345 max_conns=3; } upstream dns_servers { least_conn; server 192.168.136.130:53; server 192.168.136.131:53; server 192.168.136.132:53; } server { listen 12345; proxy_pass stream_backend; proxy_timeout 3s; proxy_connect_timeout 1s; } server { listen 53 udp; proxy_pass dns_servers; } server { listen 12346; proxy_pass backend4.example.com:12346; } } ``` In this example, all TCP and UDP proxy‑related functionality is configured inside the [`stream`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) block, just as settings for HTTP requests are configured in the [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) block. There are two named [`upstream`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) blocks, each containing three servers that host the same content as one another. In the [`server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) for each server, the server name is followed by the obligatory port number. Connections are distributed among the servers according to the [Least Connections](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#least_conn) load‑balancing method: a connection goes to the server with the fewest number of active connections. The three [`server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) blocks define three virtual servers: - The first server listens on port 12345 and proxies all TCP connections to the **stream_backend** group of upstream servers. Note that the [`proxy_pass`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_pass) directive defined in the context of the `stream` module must not contain a protocol. Two optional timeout parameters are specified: the [`proxy_connect_timeout`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout) directive sets the timeout required for establishing a connection with a server in the **stream_backend** group. The [`proxy_timeout`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) directive sets a timeout used after proxying to one of the servers in the **stream_backend** group has started. - The second server listens on port 53 and proxies all UDP datagrams (the `udp` parameter to the `listen` directive) to an upstream group called **dns_servers**. If the `udp` parameter is not specified, the socket listens for TCP connections. - The third virtual server listens on port 12346 and proxies TCP connections to **backend4.example.com**, which can resolve to several IP addresses that are load balanced with the Round Robin method. ## Compatibility note {#compat} - [TCP proxying](https://nginx.org/en/docs/stream/ngx_stream_core_module.html) and [load balancing](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html) are supported in NGINX Plus since [Release 5](nginx/releases.md#r5) and in NGINX Open Source since 1.9.0. - [UDP proxying](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#udp) and load balancing are supported in NGINX Plus since [Release 9](nginx/releases.md#r9) and in NGINX Open Source since 1.9.13. - The [`least_time`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#least_time) load-balancing method is supported in NGINX Plus since [Release 6](nginx/releases.md#r6) and in NGINX Open Source since 1.31.0. ## See also - [TCP Health Checks](tcp-health-check.md) - [UDP Health Checks](udp-health-check.md) - [Load Balancing DNS Traffic with NGINX and NGINX Plus](https://www.f5.com/company/blog/nginx/load-balancing-dns-traffic-nginx-plus) - [TCP/UDP Load Balancing with NGINX: Overview, Tips, and Tricks](https://blog.nginx.org/blog/tcp-load-balancing-udp-load-balancing-nginx-tips-tricks) -------------------------------------------------------------------------------- title: "Debugging NGINX" url: https://docs.nginx.com/nginx/admin-guide/monitoring/debugging/index.md description: Troubleshoot problems and track down bugs in an NGINX or F5 NGINX Plus deployment, with the debugging binary, debug logging, and core dumps. -------------------------------------------------------------------------------- ## Introduction Debugging helps to identify a bug in the program code if something goes wrong. It is generally used in developing or testing third-party or experimental modules. NGINX debugging features include the debugging log and creation of a core dump file with its further backtrace. ### Configuring NGINX Binary For Debugging First, you will need to enable debugging in NGINX binary. NGINX Plus already provides you with _nginx-debug_ binary while NGINX Open Source requires recompilation. ### Configuring F5 NGINX Plus Binary Starting from [Release 8](nginx/releases.md), NGINX Plus ships the _nginx-debug_ binary together with the standard binary. To enable debugging in NGINX Plus, you will need to switch from _nginx_ to _nginx-debug_ binary. Open terminal and run the command: ```shell service nginx stop && service nginx-debug start ``` When finished, [enable](#error_log) the debugging log in the configuration file. ### Compiling NGINX Open Source Binary To enable debugging in NGINX Open Source, you will need to recompile it with the `--with-debug` flag specified in the configure script. To compile NGINX Open Source with the debug support: 1. Download and unpack NGINX source files, go to the directory with the source files. See [Downloading the Sources](nginx/admin-guide/installing-nginx/installing-nginx-open-source.md). 2. Get the list of NGINX configure arguments. Run the command: ```shell nginx -V 2>&1 | grep arguments ``` 3. Add the `--with-debug` option to the list of configure commands and run the configure script: ```shell ./configure --with-debug ``` 4. Compile and install NGINX: ```shell sudo make sudo make install ``` 5. Restart NGINX. ### NGINX and Debugging Symbols Debug symbols helps obtain additional information for debugging, such as functions, variables, data structures, source file and line number information. NGINX by default is compiled with the “_-g_” flag that includes debug symbols. However, if you get the “No symbol table info available” error when you run a [backtrace](#backtrace), then debugging symbols are missing and you will need to recompile NGINX with support of debugging symbols. The exact set of compiler flags depends on the compiler. For example, for the GCC compiler system: - include debugging symbols with the “_-g_” flag - make the debugger output easier to understand by disabling compiler optimization with the “_-O0_” flag: ```shell ./configure --with-debug --with-cc-opt='-O0 -g' ... ``` ## Enabling Debug Logging in NGINX Configuration The debugging log records errors and any debugging-related information and is disabled by default. To enable it, make sure NGINX is compiled to support debugging (see [Configuring NGINX Binary For Debugging](#enable)) and then enable it in NGINX configuration file with the `debug` parameter of the [error_log](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive. The debugging log may be written to a file, an allocated _buffer_ in memory, _stderr_ output, or to _syslog_. It is recommended enabling the debugging log on the ”_main_“ level of NGINX configuration to get the full picture of what’s going on. ### Writing the Debugging Log to a File Writing the debugging log to a file may slow down performance under high load. Also note that the file can grow very large and quickly eat up disk space. To reduce the negative impact, you can configure the debugging log to be written into a memory buffer, or set the debugging log for particular IP addresses. See [Writing the Debugging Log to Memory](#error_log_memory) and [Debug Log for Selected IPs](#error_log_ip) for details. To enable writing the debugging log to a file: 1. Make sure your NGINX is configured with the `--with-debug` configuration option. Run the command and check if the output contains the `--with-debug` line: ```shell nginx -V 2>&1 | grep -- '--with-debug' ``` 2. Open NGINX configuration file: ```shell sudo vi /etc/nginx/nginx.conf ``` 3. Find the [error_log](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive which is by default located in the `main` context, and change the logging level to `debug`. If necessary, change the path to the log file: ```shell error_log /var/log/nginx/error.log debug; ``` 4. Save the configuration and exit the configuration file. ### Writing the Debugging Log to Memory The debugging log can be written to a memory using a cyclic buffer. The advantage is that logging on the debug level will not have significant impact on performance under high load. To enable writing the debug log to memory: 1. Make sure your NGINX is configured with the `--with-debug` configuration option. Run the command and check if the output contains the `--with-debug` line: ```shell nginx -V 2>&1 | grep -- '--with-debug' ``` 2. In NGINX configuration file, enable a memory buffer for debug logging with the [error_log](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive specified in the `main` context: ```nginx error_log memory:32m debug; ... http { ... } ``` #### Extracting Debug Log From Memory The log can be extracted from the memory buffer using a script executed in the GDB debugger. To extract the debugging log from memory: 1. Obtain the PID of NGINX worker process: ```shell ps axu |grep nginx ``` 2. Launch the GDB debugger: ```shell sudo gdb -p ``` 3. Copy the script, paste it to GDB and press “Enter”. The script will save the log in the _debug_log.txt_ file located in the current directory: ```nginx set $log = ngx_cycle->log while $log->writer != ngx_log_memory_writer set $log = $log->next end set $buf = (ngx_log_memory_buf_t *) $log->wdata dump binary memory debug_log.txt $buf->start $buf->end ``` 4. Quit GDB by pressing CTRL+D. 5. Open the file “_debug_log.txt_” located in the current directory: ```shell sudo less debug_log.txt ``` ### Debug Log for Selected IPs It is possible to enable the debugging log for a particular IP address or a range of IP addresses. Logging particular IPs may useful in a production environment as it will not negatively affect performance. The IP address is specified in the [debug_connection](https://nginx.org/en/docs/ngx_core_module.html#debug_connection) directive within the [events](https://nginx.org/en/docs/ngx_core_module.html#events) block; the directive can be defined more than once: ```nginx error_log /path/to/log; ... events { debug_connection 192.168.1.1; debug_connection 192.168.10.0/24; } ``` ### Debug Log for Each Virtual Host Generally, the [error_log](https://nginx.org/en/docs/ngx_core_module.html#error_log) directive is specified in the `main` context and thus is applied to all other contexts including [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) and [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location). But if there is another `error_log` directive specified inside a particular `server` or a `location` block, the global settings will be overridden and such `error_log` directive will set its own path to the log file and the level of logging. To set up the debugging log for a particular virtual host, add the `error_log` directive inside a particular `server` block, in which set a new path to the log file and the `debug` logging level: ```nginx error_log /path1/to/log debug; ... http { ... server { error_log /path2/to/log debug; ... } } ``` To disable the debugging log per a particular virtual host, specify the `error_log` directive inside a particular `server` block, and specify a path to the log file only: ```nginx error_log /path/to/log debug; ... http { ... server { error_log /path/to/log; ... } } ``` ## Enabling Core Dumps A core dump file can help identify and fix problems that are causing NGINX to crash. A core dump file may contain sensitive information such as passwords and private keys, so ensure that they are treated securely. In order to create a core dump file, they must be enabled in both the operating system and the NGINX configuration file. ### Enabling Core Dumps in the Operating System Perform the following steps in your operating system: 1. Specify a working directory in which a core dump file will be saved, for example, “_/tmp/cores_”: ```shell mkdir /tmp/cores ``` 2. Make sure the directory is writable by NGINX worker process: ```shell sudo chown root:root /tmp/cores sudo chmod 1777 /tmp/cores ``` 3. Disable the limit for the maximum size of a core dump file: ```shell sudo prlimit --core=unlimited:unlimited --pid $(cat /run/nginx.pid) ``` If the operation ends up with “Cannot modify limit: operation not permitted”, run the command: ```shell sudo sh -c "ulimit -c unlimited && exec su $LOGNAME" ``` 4. Enable core dumps for the _setuid_ and _setgid_ processes. For CentOS 7.0, Debian 8.2, Ubuntu 14.04, run the commands: ```shell echo "/tmp/cores/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern sudo sysctl -w fs.suid_dumpable=2 sysctl -p ``` For FreeBSD, run the commands: ```shell sudo sysctl kern.sugid_coredump=1 sudo sysctl kern.corefile=/tmp/cores/%N.core.%P ``` ### Enabling Core Dumps in NGINX Configuration To enable core dumps in the NGINX configuration file: 1. Open the NGINX configuration file: ```shell sudo vi /usr/local/etc/nginx/nginx.conf ``` 2. Define a directory that will keep core dump files with the [working_directory](https://nginx.org/en/docs/ngx_core_module.html#working_directory) directive. The directive is specified on the _main_ configuration level: ```nginx working_directory /tmp/cores/; ``` 3. Make sure the directory exists and is writable by NGINX worker process. Open terminal and run the commands: ```shell sudo chown root:root /tmp/cores sudo chmod 1777 /tmp/cores ``` 4. Specify the maximum possible size of the core dump file with the [worker_rlimit_core](https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_core) directive. The directive is also specified on the `main` configuration level. If the core dump file size exceeds the value, the core dump file will not be created. ```nginx worker_rlimit_core 500M; ``` Example: ```nginx worker_processes auto; error_log /var/log/nginx/error.log debug; working_directory /tmp/cores/; worker_rlimit_core 500M; events { ... } http { ... } ``` With these settings, a core dump file will be created in the “_/tmp/cores/_” directory, and only if its size does not exceed 500 megabytes. ## Obtaining Backtrace From a Core Dump File Backtraces provide information from a core dump file about what was wrong when a program crashed. To get a backtrace from a core dump file: 1. Open a core dump file with the GDB debugger using the pattern: ```shell sudo gdb ``` 2. Type-in the “_backtrace_ command to get a stack trace from the time of the crash: ```nginx (gdb) backtrace ``` If the “_backtrace_” command resulted with the “No symbol table info available” message, you will need to recompile NGINX binary to include debugging symbols. See [NGINX and Debugging Symbols](#compile_symbols). ## Dumping NGINX Configuration From a Running Process You can extract the current NGINX configuration from the master process in memory. This can be useful when you need to: - verify which configuration has been loaded - restore a previous configuration if the version on disk has been accidentally removed or overwritten The configuration dump can be obtained with a GDB script provided that your NGINX has the debug support. 1. Make sure your NGINX is built with the debug support (the `--with-debug` configure option in the list of the configure arguments). Run the command and check if the output contains the `--with-debug` line: ```shell nginx -V 2>&1 | grep -- '--with-debug' ``` 2. Obtain the PID of NGINX worker process: ```shell ps axu | grep nginx ``` 3. Launch the GDB debugger: ```shell sudo gdb -p ``` 4. Copy and paste the script to GDB and press “Enter”. The script will save the configuration in the _nginx_conf.txt_ file in the current directory: ```nginx set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf "Dumping %s to nginx_conf.txt\n", $name append memory nginx_conf.txt \ $elts[$nelts]->buffer.start $elts[$nelts]->buffer.end end ``` 5. Quit GDB by pressing _CTRL+D_. 6. Open the file _nginx_conf.txt_ located in the current directory: ```shell sudo vi nginx.conf.txt ``` ## Asking for help When asking for help with debugging, please provide the following information: 1. NGINX version, compiler version, and configure parameters. Run the command: ```shell nginx -V ``` 2. Current full NGINX configuration. See [Dumping NGINX Configuration From a Running Process](#configdump) 3. The debugging log. See [Enabling Debug Logging in NGINX Configuration](#error_log) 4. The obtained backtrace. See [Enabling Core Dumps](#coredump), [Obtaining Backtrace](#backtrace) -------------------------------------------------------------------------------- title: "HTTP Health Checks" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/index.md description: Monitor the health of HTTP servers in an upstream group by sending periodic health checks, including customizable active health checks in F5 NGINX Plus. -------------------------------------------------------------------------------- ## Introduction NGINX and F5 NGINX Plus can continually test your upstream servers, avoid the servers that have failed, and gracefully add the recovered servers into the load‑balanced group. ## Prerequisites - For passive health checks, [NGINX Open Source](https://nginx.org/en/) or [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) - For active health checks and the [live activity monitoring dashboard](nginx/admin-guide/monitoring/live-activity-monitoring.md), NGINX Plus - A load‑balanced group of [HTTP upstream servers](http-load-balancer.md) ## Passive Health Checks For passive health checks, NGINX and NGINX Plus monitor transactions as they happen, and try to resume failed connections. If the transaction still cannot be resumed, NGINX Open Source and NGINX Plus mark the server as unavailable and temporarily stop sending requests to it until it is marked active again. The conditions under which an upstream server is marked unavailable are defined for each upstream server with parameters to the [`server`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#server) directive in the `upstream` block: - [`fail_timeout`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) – Sets the time during which a number of failed attempts must happen for the server to be marked unavailable, and also the time for which the server is marked unavailable (default is 10 seconds). - [`max_fails`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) – Sets the number of failed attempts that must occur during the `fail_timeout` period for the server to be marked unavailable (default is 1 attempt). In the following example, if NGINX fails to send a request to a server or does not receive a response from it 3 times in 30 seconds, it marks the server as unavailable for 30 seconds: ```nginx upstream backend { server backend1.example.com; server backend2.example.com max_fails=3 fail_timeout=30s; } ``` Note that if there is only a single server in a group, the `fail_timeout` and `max_fails` parameters are ignored and the server is never marked unavailable. ### Server Slow Start A recently recovered server can be easily overwhelmed by connections, which may cause the server to be marked as unavailable again. Slow start allows an upstream server to gradually recover its weight from zero to its nominal value after it has been recovered or became available. This can be done with the [`slow_start`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start) parameter of the upstream [`server`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#match) directive: ```nginx upstream backend { server backend1.example.com slow_start=30s; server backend2.example.com; server 192.0.0.1 backup; } ``` Note that if there is only a single server in a group, the `slow_start` parameter is ignored and the server is never marked unavailable. Slow start is exclusive to NGINX Plus. ## Active Health Checks NGINX Plus can periodically check the health of upstream servers by sending special health‑check requests to each server and verifying the correct response. To enable active health checks: 1. In the [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that passes requests ([`proxy_pass`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass)) to an upstream group, include the [`health_check`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive: ```nginx server { location / { proxy_pass http://backend; health_check; } } ``` This snippet defines a server that passes all requests (`location /`) to the upstream group called `backend`. It also enables advanced health monitoring with the [`health_check`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive: by default, every five seconds NGINX Plus sends a request for "**/**" to each server in the `backend` group. If any communication error or timeout occurs (the server responds with a status code outside the range from `200` through `399`) the health check fails. The server is marked as unhealthy, and NGINX Plus does not send client requests to it until it once again passes a health check. Optionally you can specify another port for health checks, for example, for monitoring health of many services on the same host. Specify a new port with the [`port`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_port) parameter of the [`health_check`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive: ```nginx server { location / { proxy_pass http://backend; health_check port=8080; } } ``` 2. In the upstream server group, define a shared memory zone with the [`zone`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) directive: ```nginx http { upstream backend { zone backend 64k; server backend1.example.com; server backend2.example.com; server backend3.example.com; server backend4.example.com; } } ``` The zone is shared among all worker processes and stores the configuration of the upstream group. This [enables](/nginx/admin-guide/load-balancer/http-load-balancer.md#sharing-data-with-multiple-worker-processes) the worker processes to use the same set of counters to keep track of responses from the servers in the group. The defaults for active health checks can be overridden with parameters to the `health_check` directive: ```nginx location / { proxy_pass http://backend; health_check interval=10 fails=3 passes=2; } ``` Here, the [`interval`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_interval) parameter increases the delay between health checks from the default 5 seconds to 10 seconds. The [`fails`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_fails) parameter requires the server to fail three health checks to be marked as unhealthy (up from the default one). Finally, the [`passes`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_passes) parameter means the server must pass two consecutive checks to be marked as healthy again instead of the default one. You can also enable connection caching with the [`keepalive_time`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_keepalive_time) parameter - in case of TLS upstreams the full TLS handshake won't happen for every health check probe and the connection can be reused during the specified period of time: ```nginx location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass https://backend; health_check interval=1 keepalive_time=60s; } ``` ### Specifying the Requested URI Use the [`uri`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_uri) parameter of the [`health_check`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive to set the URI to request in a health check: ```nginx location / { proxy_pass http://backend; health_check uri=/some/path; } ``` The specified URI is appended to the server domain name or IP address set for the server in the `upstream` block. For the first server in the sample `backend` group declared above, a health check requests the URI **"http://backend1.example.com/some/path"**. ### Defining Custom Conditions You can set custom conditions that the response must satisfy for the server to pass the health check. The conditions are defined in a [`match`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#match) block, which is referenced in the [`match`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_match) parameter of the [`health_check`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive. 1. On the `http {}` level, specify the `match` `{}` block and name it, for example, `server_ok`: ```nginx http { #... match server_ok { # tests are here } } ``` 2. Refer to the block from the [`health_check`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive by specifying the [`match`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_match) parameter and the name of the [`match`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#match) block: ```nginx http { #... match server_ok { status 200-399; body !~ "maintenance mode"; } server { #... location / { proxy_pass http://backend; health_check match=server_ok; } } } ``` Here the health check is passed if the status code of the response is in the range `200`–`399`, and its body does not contain the string `maintenance mode`. The `match` directive enables NGINX Plus to check the status code, header fields, and the body of a response. Using this directive it is possible to verify whether the status is in a specified range, whether a response includes a header, or whether the header or body matches a regular expression. The `match` directive can contain one status condition, one body condition, and multiple header conditions. A response must satisfy all conditions defined in `match` block for the server to pass the health check. For example, the following `match` directive matches responses that have status code `200`, the exact value `text/html` in the `Content-Type` header, and the text `Welcome to nginx!` in the body: ```nginx match welcome { status 200; header Content-Type = text/html; body ~ "Welcome to nginx!"; } ``` The following example uses the exclamation point (`!`) to define characteristics the response must not have to pass the health check. In this case, the health check passes when the status code is something other than `301`, `302`, `303`, or `307`, and there is no `Refresh` header. ```nginx match not_redirect { status ! 301-303 307; header ! Refresh; } ``` ### Mandatory Health Checks By default, when a new server is added to an upstream group, NGINX Plus considers it healthy and sends traffic to it immediately. But for some servers, particularly if they were added through the API interface or through DNS resolution, it would be good to perform health check first before allowing them to handle traffic. The [`mandatory`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_mandatory) parameter requires every newly added server to pass all configured health checks before NGINX Plus sends traffic to it. When combined with [`slow start`](#slow_start), it gives a new server more time to connect to databases and “warm up” before being asked to handle their full share of traffic. Mandatory health checks can be marked as persistent, so that the previous state is remembered when reloading configuration. Specify the [`persistent`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_persistent) parameter together with the [`mandatory`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_mandatory) parameter: ```nginx upstream my_upstream { zone my_upstream 64k; server backend1.example.com slow_start=30s; } server { location / { proxy_pass http://my_upstream; health_check mandatory persistent; } } ``` Here the `mandatory` and `persistent` parameters of the `health_check` directive and the `slow_start` parameter of the `server` directive are specified. Servers that are added to the upstream group using the API or DNS interfaces are marked as unhealthy and receive no traffic until they pass the health check; at that point they start receiving a gradually increasing amount of traffic over a span of 30 seconds. If NGINX Plus configuration is reloaded and before reload the server was marked as healthy, mandatory health check are not performed and the server state is considered to be `up`. Health checks can also be enabled for non-HTTP protocols, such as [FastCGI](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html), [memcached](https://nginx.org/en/docs/http/ngx_http_memcached_module.html), [SCGI](https://nginx.org/en/docs/http/ngx_http_scgi_module.html), [uwsgi](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html), and also for [TCP and UDP](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check). -------------------------------------------------------------------------------- title: "Installing NGINX Plus AMIs on Amazon EC2" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus-amazon-web-services/index.md description: Install F5 NGINX Plus on Amazon Web Services (AWS), to provide sophisticated Layer 7 load balancing for your apps running on Amazon Linux, RHEL, and Ubuntu. -------------------------------------------------------------------------------- NGINX, Inc. participates in the Amazon Web Services (AWS) Partner Network as a Standard Technology Partner. We offer Amazon Machine Images (AMIs) for use in the Amazon Elastic Compute Cloud (EC2), available at the AWS Marketplace for several operating systems, including Amazon Linux, Red Hat Enterprise Linux, and Ubuntu. The AMIs contain the following components: - Latest version of [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus), optimized for use on Amazon EC2 - Pre-packaged software for building highly available (HA) NGINX Plus configurations ## Install the F5 NGINX Plus AMI To quickly set up an NGINX Plus environment on AWS: 1. Follow the instructions in [Getting Started with Amazon EC2 Linux Instances](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) to sign up on AWS and get more information about EC2 itself. 2. Proceed to the product page for the appropriate AMI at the AWS Marketplace, and launch the AMI. - [NGINX Plus – Amazon Linux AMI HVM](https://aws.amazon.com/marketplace/seller-profile?id=741df81b-dfdc-4d36-b8da-945ea66b522c) - [NGINX Plus – Red Hat Enterprise Linux 7 AMI HVM](https://aws.amazon.com/marketplace/seller-profile?id=741df81b-dfdc-4d36-b8da-945ea66b522c) - [NGINX Plus – Red Hat Enterprise Linux 6 AMI HVM](https://aws.amazon.com/marketplace/seller-profile?id=741df81b-dfdc-4d36-b8da-945ea66b522c) - [NGINX Plus – Ubuntu AMI HVM](https://aws.amazon.com/marketplace/seller-profile?id=741df81b-dfdc-4d36-b8da-945ea66b522c) Click the **Continue to Subscribe** button to proceed to the **Launch on EC2** page. 3. Select the type of launch by clicking the appropriate tab (**1-Click Launch**, **Manual Launch**, or **Service Catalog**). Choose the desired options for billing, instance size, and so on, and click the **Accept Software Terms…** button. 4. When configuring the firewall rules, add a rule to accept web traffic on TCP ports 80 and 443 (this happens automatically if you launch from the **1-Click Launch** tab). 5. As soon as the new EC2 instance launches, NGINX Plus starts automatically and serves a default **index.html** page. To view the page, use a web browser to access the public DNS name of the new instance. You can also check the status of the NGINX Plus server by logging into the EC2 instance and running this command: ```nginx /etc/init.d/nginx status ``` See [NGINX Plus on the AWS Cloud Quick Start](https://aws.amazon.com/about-aws/whats-new/2017/08/quick-start-update-deploy-nginx-plus-on-the-aws-cloud/) deployment guide for details. ## What If I Need Help? If you encounter any problems with NGINX Plus configuration, documentation is available at [nginx.org](https://nginx.org/en/docs/) and in the [NGINX Plus Admin Guide](/nginx/admin-guide/installing-nginx/). Customers who purchase an NGINX Plus AMI at the AWS Marketplace are eligible for the AWS support provided by the NGINX, Inc. engineering team. To activate support, submit the [AMI Support Activation](https://www.nginx.com/ami-support-activation/) form (you need your AWS account number). When you request support, we’ll ask you to provide the AWS account number that you registered, along with the IDs of your EC2 instances in some cases. -------------------------------------------------------------------------------- title: "NGINX Reverse Proxy" url: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/index.md description: Configure NGINX as a reverse proxy for HTTP and other protocols, with support for modifying request headers and fine-tuned buffering of responses. -------------------------------------------------------------------------------- This article describes the basic configuration of a proxy server. You will learn how to pass a request from NGINX to proxied servers over different protocols, modify client request headers that are sent to the proxied server, and configure buffering of responses coming from the proxied servers. ## Introduction Proxying is typically used to distribute the load among several servers, seamlessly show content from different websites, or pass requests for processing to application servers over protocols other than HTTP. ## Passing a Request to a Proxied Server When NGINX proxies a request, it: 1. Sends the request to a specified proxy server 1. Fetches the response 1. Sends the response back to the client. It is possible to proxy requests to an HTTP server (another NGINX server or any other server) or a non-HTTP server (which can run an application developed with a specific framework, such as PHP or Python) using a specified protocol. Supported protocols include [FastCGI](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html), [uwsgi](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html), [SCGI](https://nginx.org/en/docs/http/ngx_http_scgi_module.html), and [memcached](https://nginx.org/en/docs/http/ngx_http_memcached_module.html). To pass a request to an HTTP proxied server, the [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive is specified inside a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location). For example: ```nginx location /some/path/ { proxy_pass http://www.example.com/link/; } ``` This example configuration results in passing all requests processed in this location to the proxied server at the specified address. This address can be specified as a domain name or an IP address. The address may also include a port: ```nginx location ~ \.php { proxy_pass http://127.0.0.1:8000; } ``` Note that in the first example above, the address of the proxied server is followed by a URI, `/link/`. If the URI is specified along with the address, it replaces the part of the request URI that matches the location parameter. For example, the request with the `/some/path/page.html` URI will be proxied to `http://www.example.com/link/page.html`. However, if the address is specified without a URI, or it is not possible to determine the part of URI to be replaced, the full request URI is passed (possibly, modified). To pass a request to a non-HTTP proxied server, the appropriate `**_pass` directive should be used: - [fastcgi_pass](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_pass) passes a request to a FastCGI server - [uwsgi_pass](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html#uwsgi_pass) passes a request to a uwsgi server - [scgi_pass](https://nginx.org/en/docs/http/ngx_http_scgi_module.html#scgi_pass) passes a request to an SCGI server - [memcached_pass](https://nginx.org/en/docs/http/ngx_http_memcached_module.html#memcached_pass) passes a request to a memcached server Note that in these cases, the rules for specifying addresses may be different. You may also need to pass additional parameters to the server (see the [reference documentation](https://nginx.org/en/docs/) for more detail). The [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive can also be used to pass requests to a [named group](https://nginx.org/en/docs/http/load_balancing.html#algorithms) of servers. In this case, requests are distributed among the servers in the group according to the [specified method](https://www.nginx.com/resources/admin-guide/load-balancer/). ## Passing Request Headers By default, NGINX modifies two header fields in proxied requests, “Host” and “Connection”, and eliminates the header fields whose values are empty strings. “Host” is set to the value of the `$proxy_host` variable, and “Connection” is set to `close`. To change these settings, as well as modify other header fields, use the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. This directive can be specified in a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) or higher. It can also be specified in a particular [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) context or in the [http](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) block. For example: ```nginx location /some/path/ { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://localhost:8000; } ``` In this configuration the “Host” field is set to the [$host](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables) variable. To prevent a header field from being passed to the proxied server, set it to an empty string as follows: ```nginx location /some/path/ { proxy_set_header Accept-Encoding ""; proxy_pass http://localhost:8000; } ``` ## Configuring Buffers By default NGINX buffers responses from proxied servers. A response is stored in the internal buffers and is not sent to the client until the whole response is received. Buffering helps to optimize performance with slow clients, which can waste proxied server time if the response is passed from NGINX to the client synchronously. However, when buffering is enabled NGINX allows the proxied server to process responses quickly, while NGINX stores the responses for as much time as the clients need to download them. The directive that is responsible for enabling and disabling buffering is [proxy_buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering). By default it is set to `on` and buffering is enabled. The [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) and [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) directives control how NGINX stores and buffers data. The [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive controls the size and the number of buffers allocated for a request. The first part of the response from a proxied server is stored in a separate buffer, the size of which is set with the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) directive. This part usually contains a comparatively small response header and can be made smaller than the buffers for the rest of the response. In the following example, the default number of buffers is increased and the size of the buffer for the first portion of the response is made smaller than the default. ```nginx location /some/path/ { proxy_buffers 16 4k; proxy_buffer_size 2k; proxy_pass http://localhost:8000; } ``` If buffering is disabled, the response is sent to the client synchronously while it is receiving it from the proxied server. This behavior may be desirable for fast interactive clients that need to start receiving the response as soon as possible. To disable buffering in a specific location, place the [proxy_buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) directive in the [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with the `off` parameter, as follows: ```nginx location /some/path/ { proxy_buffering off; proxy_pass http://localhost:8000; } ``` In this case NGINX uses only the buffer configured by [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) to store the current part of a response. A common use of a reverse proxy is to provide load balancing. See [HTTP Load Balancing](nginx/admin-guide/load-balancer/http-load-balancer.md) and [TCP and UDP Load Balancing](nginx/admin-guide/load-balancer/tcp-udp-load-balancer.md). ## Choosing an Outgoing IP Address If your proxy server has several network interfaces, sometimes you might need to choose a particular source IP address for connecting to a proxied server or an upstream. This may be useful if a proxied server behind NGINX is configured to accept connections from particular IP networks or IP address ranges. In such cases, you can specify the [proxy_bind](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_bind) directive and the IP address of the necessary network interface: ```nginx location /app1/ { proxy_bind 127.0.0.1; proxy_pass http://example.com/app1/; } location /app2/ { proxy_bind 127.0.0.2; proxy_pass http://example.com/app2/; } ``` The IP address can be also specified with a variable. For example, the [`$server_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_server_addr) variable passes the IP address of the network interface that accepted the request: ```nginx location /app3/ { proxy_bind $server_addr; proxy_pass http://example.com/app3/; } ``` -------------------------------------------------------------------------------- title: "Restricting Access with HTTP Basic Authentication" url: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/index.md description: Control access using HTTP Basic authentication, and optionally in combination with IP address-based access control. -------------------------------------------------------------------------------- ## Introduction You can restrict access to your website or some parts of it by implementing a username/password authentication. Usernames and passwords are taken from a file created and populated by a password file creation tool, for example, `apache2-utils`. HTTP Basic authentication can also be combined with other access restriction methods, for example restricting access by [IP address](denylisting-ip-addresses.md) or [geographical location](controlling-access-by-geoip.md). ## Prerequisites - F5 NGINX Plus or NGINX Open Source - Password file creation utility such as `apache2-utils` (Debian, Ubuntu) or `httpd-tools` (RHEL/CentOS/Oracle Linux). ## Creating a Password File To create username-password pairs, use a password file creation utility, for example, `apache2-utils` or `httpd-tools` 1. Verify that `apache2-utils` (Debian, Ubuntu) or `httpd-tools` (RHEL/CentOS/Oracle Linux) is installed. 2. Create a password file and a first user. Run the `htpasswd` utility with the `-c` flag (to create a new file), the file pathname as the first argument, and the username as the second argument: ```shell sudo htpasswd -c /etc/apache2/.htpasswd user1 ``` Press Enter and type the password for **user1** at the prompts. 3. Create additional user-password pairs. Omit the `-c` flag because the file already exists: ```shell sudo htpasswd /etc/apache2/.htpasswd user2 ``` 4. You can confirm that the file contains paired usernames and hashed passwords: ```shell $ cat /etc/apache2/.htpasswd user1: user2: user3: ``` ## Configuring NGINX and NGINX Plus for HTTP Basic Authentication 1. Inside a location that you are going to protect, specify the [auth_basic](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html#auth_basic) directive and give a name to the password-protected area. The name of the area will be shown in the username/password dialog window when asking for credentials: ```nginx location /api { auth_basic "Administrator’s Area"; #... } ``` 2. Specify the [auth_basic_user_file](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html#auth_basic_user_file) directive with a path to the _.htpasswd_ file that contain user/password pairs: ```nginx location /api { auth_basic "Administrator’s Area"; auth_basic_user_file /etc/apache2/.htpasswd; } ``` Alternatively, you you can limit access to the whole website with basic authentication but still make some website areas public. In this case, specify the `off` parameter of the [auth_basic](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html#auth_basic) directive that cancels inheritance from upper configuration levels: ```nginx server { ... auth_basic "Administrator’s Area"; auth_basic_user_file conf/htpasswd; location /public/ { auth_basic off; } } ``` ## Combining Basic Authentication with Access Restriction by IP Address HTTP basic authentication can be effectively combined with access restriction by IP address. You can implement at least two scenarios: - a user must be both authenticated and have a valid IP address - a user must be either authenticated, or have a valid IP address 1. Allow or deny access from particular IP addresses with the [allow](https://nginx.org/en/docs/http/ngx_http_access_module.html#allow) and [deny](https://nginx.org/en/docs/http/ngx_http_access_module.html#deny) directives: ```nginx location /api { #... deny 192.168.1.2; allow 192.168.1.1/24; allow 127.0.0.1; deny all; } ``` Access will be granted only for the `192.168.1.1/24` network excluding the `192.168.1.2` address. Note that the `allow` and `deny` directives will be applied in the order they are defined. 2. Combine restriction by IP and HTTP authentication with the [satisfy](https://nginx.org/en/docs/http/ngx_http_core_module.html#satisfy) directive. If you set the directive to `all`, access is granted if a client satisfies both conditions. If you set the directive to `any`, access is granted if if a client satisfies at least one condition: ```nginx location /api { #... satisfy all; deny 192.168.1.2; allow 192.168.1.1/24; allow 127.0.0.1; deny all; auth_basic "Administrator’s Area"; auth_basic_user_file conf/htpasswd; } ``` ## Complete Example The example shows how to protect your status area with simple authentication combined with access restriction by IP address: ```nginx http { server { listen 192.168.1.23:8080; root /usr/share/nginx/html; location /api { api; satisfy all; deny 192.168.1.2; allow 192.168.1.1/24; allow 127.0.0.1; deny all; auth_basic "Administrator’s Area"; auth_basic_user_file /etc/apache2/.htpasswd; } } } ``` When you access your status page, you are prompted to log in: ![The "Authentication Required" prompt](/nginx/images/auth-required.png) If the provided name and password do not match the password file, you get the `401 (Authorization Required)` error. -------------------------------------------------------------------------------- title: "Single Sign-On with Microsoft Active Directory FS" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/active-directory-federation-services/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Microsoft AD FS as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Microsoft Active Directory Federation Services](https://docs.microsoft.com/en-us/windows-server/identity/active-directory-federation-services) (AD FS) as the Identity Provider (IdP) and NGINX Plus as the Relying Party (RP), or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. In the latest NGINX Plus version, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - A Microsoft AD FS instance, either on-premises or in [Azure](https://learn.microsoft.com/en-us/windows-server/identity/ad-fs/deployment/how-to-connect-fed-azure-adfs), with administrator privileges. - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure the AD FS Server {#adfs-setup} [Microsoft Active Directory Federation Services](https://docs.microsoft.com/en-us/windows-server/identity/active-directory-federation-services) (AD FS) serves as the Identity Provider. ### Create an AD FS Application 1. In AD FS, open the Server Manager. 2. In Server Manager, select **Tools**, and then select **AD FS Management**. 3. In **AD FS Management**, right-click on **Application Groups** and select **Add Application Group**. 4. On the Application Group Wizard **Welcome** screen: - Enter the Name of your application, for example, `NGINX Demo App`. - Under **Standalone applications**, select **Server application**. 5. On the Application Group Wizard **Server application** screen: - Copy the **Client Identifier** value generated by AD FS. The client identifier is your AD FS Application ID, you will need it later when configuring NGINX Plus. - In **Redirect URI**, enter the Redirect URI for your NGINX Plus instance, for example, `https://demo.example.com/oidc_callback`, and then click **Add**. 6. On the Application Group Wizard **Configure Application Credentials** screen: - Select **Generate a shared secret**. - Copy and save the generated **Client Secret**, you will need it later when configuring NGINX Plus. You will not be able to view the secret after the application group is created. - Select **Next** to complete the steps for adding the application group. ### Configure Logout URLs After creating the application group, you need to configure the logout URLs to support RP-initiated logout: 1. In **AD FS Management**, navigate to **Application Groups** and select your application group. 2. Right-click on the application group and select **Properties**. 3. In the **Properties** dialog, add the post logout redirect URI to the application configuration: - Add the post logout URL, for example: `https://demo.example.com/post_logout/`. 4. To enable OpenID Connect front-channel logout (single sign-out when the user signs out of another application): - Use PowerShell to configure a **LogoutUri** for the AD FS client (there is no GUI option for this). For example, run: ```powershell Set-AdfsClient -TargetClientId -LogoutUri https://demo.example.com/front_logout/ ``` Replace `` with the Client Identifier from [Step 5](#adfs-setup-id) above (the AD FS Application ID) and substitute the domain name of your NGINX Plus instance for `demo.example.com`. This registers a front-channel logout URL (`LogoutUri`) for the client in AD FS. When a user signs out of this or any other application in AD FS, the AD FS server sends a GET request to this URL (typically via a hidden iframe) with the user's session ID (`sid`) as a query parameter, instructing NGINX Plus to clear the user's session. According to the OpenID Connect front-channel logout specification, the identity provider is supposed to send both an issuer (`iss`) and a session ID; AD FS provides only the `sid` parameter, but the NGINX Plus OIDC module supports both the fully compliant `iss+sid` and the `sid`-only variants and will clear the session in either case. ### Get the OpenID Connect Discovery URL Check the OpenID Connect endpoint URL. By default, AD FS publishes the `.well-known/openid-configuration` document at the following address: `https://adfs-server-address/adfs/.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https://adfs-server-address/adfs/.well-known/openid-configuration | jq . ``` Where: - the `adfs-server-address` is your AD FS server address - the `/adfs/.well-known/openid-configuration` is the default address for AD FS for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://adfs-server-address/adfs", "authorization_endpoint": "https://adfs-server-address/adfs/oauth2/authorize/", "token_endpoint": "https://adfs-server-address/adfs/oauth2/token/", "jwks_uri": "https://adfs-server-address/adfs/discovery/keys", "userinfo_endpoint": "https://adfs-server-address/adfs/userinfo", "end_session_endpoint": "https://adfs-server-address/adfs/oauth2/logout", "frontchannel_logout_supported": true, "frontchannel_logout_session_supported": true ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for AD FS is: `https://adfs-server-address/adfs`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ## Set up NGINX Plus {#nginx-plus-setup} With AD FS configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```text nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [AD FS Configuration](#adfs-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive. By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the AD FS OIDC provider named `adfs` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider adfs { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - Your actual AD FS **Client ID** from [Step 5](#adfs-setup-id) of AD FS Configuration with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - Your **Client Secret** from [Step 6](#adfs-setup-secret) of AD FS Configuration with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - The **Issuer** URL from [Step 2](#adfs-setup-issuer) of AD FS Configuration with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your AD FS OIDC URL. By default, NGINX forms the provider metadata endpoint by appending `.well-known/openid-configuration` to the issuer. For AD FS, this often resolves to `https://adfs-server-address/adfs/.well-known/openid-configuration`. If your AD FS issuer differs from `https://adfs-server-address/adfs` (for example, a custom path), you can explicitly specify the metadata document with the [`config_url`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url) directive. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **frontchannel_logout_uri** directive defines the URI that receives OpenID Connect front-channel logout requests from AD FS. This URI must be an HTTPS path and must match the LogoutUri configured for the client in AD FS. When AD FS triggers a front-channel logout (for example, when a user signs out of another application), it sends a GET request to this URI (typically via a hidden iframe) with the session ID (sid) as a query parameter. The OIDC module clears the corresponding user session on NGINX Plus. - The **post_logout_uri** is absolute HTTPS URL where AD FS should redirect the user after a successful logout. This value **must also be configured** in the AD FS application properties. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user's ID token as a *hint* to AD FS. This directive is **optional**, however, if it is omitted the AD FS may display an extra confirmation page asking the user to approve the logout request. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch `/userinfo` from the AD FS and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when the provider's OpenID Connect discovery document advertises the S256 code challenge method in the code_challenge_methods_supported field. You can override this behavior with the pkce directive: set `pkce off;` to disable PKCE even when S256 is advertised, or `pkce on;` to force PKCE even if the IdP's metadata does not list S256. - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system's CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP's certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP's certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider adfs { issuer https://adfs.example.com/adfs; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; frontchannel_logout_uri /front_logout; logout_token_hint on; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with AD FS OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `adfs` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#adfs-setup-oidc-provider): ```nginx # ... location / { auth_oidc adfs; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by AD FS: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by AD FS - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc adfs; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nAD FS sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider adfs { # The 'issuer' is typically your AD FS OIDC URL # e.g. https://adfs.example.com/adfs issuer https://adfs.example.com/adfs; # Replace with your actual AD FS Client ID and Secret client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Front-channel logout frontchannel_logout_uri /front_logout; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration (enabled automatically when supported by the IdP) # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Protect this location with AD FS OIDC auth_oidc adfs; # Forward OIDC claims as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nAD FS sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open `https://demo.example.com/` in a browser. You will be automatically redirected to the AD FS sign-in page. 2. Enter valid AD FS credentials of a user who has access the application. Upon successful sign-in, AD FS redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; AD FS ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to AD FS for a fresh sign‑in, proving the session has been terminated. ## Legacy njs-based AD FS Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based Microsoft AD FS Guide](nginx/deployment-guides/single-sign-on/oidc-njs/active-directory-federation-services.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added front-channel logout support (`frontchannel_logout_uri`), PKCE configuration (`pkce` directive), and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "Synchronizing NGINX Configuration in a Cluster" url: https://docs.nginx.com/nginx/admin-guide/high-availability/configuration-sharing/index.md description: Sychronize configuration across a cluster of F5 NGINX Plus instances. Though designed for high-availability deployments, the solution works for any cluster. -------------------------------------------------------------------------------- ## Overview NGINX Plus is often deployed in a high‑availability (HA) cluster of two or more devices. The configuration sharing feature enables you to push configuration from one machine in the cluster (the primary) to its peers: ![nginx-sync.sh](/nginx/images/nginx-plus-config-synchronization.png) To configure this feature: 1. [Install](#in_detail1) the **nginx-sync** package on the primary machine 2. [Grant](#in_detail2) the primary machine ssh access as `root` to the peer machines 3. [Create](#in_detail3) the configuration file **/etc/nginx-sync.conf** on the primary machine: ```none NODES="node2.example.com node3.example.com node4.example.com" CONFPATHS="/etc/nginx/nginx.conf /etc/nginx/conf.d" EXCLUDE="default.conf" ``` 4. [Run](#in_detail4) the `nginx-sync.sh` command on the primary node to push the configuration files name in `CONFPATHS` to the specified `NODES`, omitting configuration files named in `EXCLUDE`. `nginx-sync.sh` includes a number of safety checks: - Verifies system prerequisites before proceeding - Validates the local (primary) configuration (`nginx -t`) and exits if that fails - Creates remote backup of the configuration on each peer - Pushes the primary configuration to the peers using `rsync`, validates configuration on the peers (`nginx -t`), and if successful reloads NGINX Plus on the peers (`service nginx reload`) - If any step fails, rolls back to the backup on the peers ## Instructions ### Installing nginx-sync on the Primary Machine Install the NGINX Synchronization module package `nginx-sync` on the Primary machine. Check the [Technical Specifications](nginx/technical-specs.md) page to verify that the module is supported by your operating system. - For Amazon Linux 2, CentOS, Oracle Linux, and RHEL: ```shell sudo yum install nginx-sync ``` - For Amazon Linux 2023, AlmaLinux, Rocky Linux: ```shell sudo dnf install nginx-sync ``` - For Ubuntu or Debian: ```shell sudo apt-get install nginx-sync ``` - For SLES: ```shell sudo zypper install nginx-sync ``` ### Configuring root SSH Access to the Peers This procedure enables the `root` user on the primary node to ssh to the `root` account on each peer, which is required to `rsync` files to the peers and run commands on the peers to validate the configuration, reload NGINX Plus, and so on. 1. On the primary node, generate an SSH authentication key pair for `root` and view the public part of the key: ```shell sudo ssh-keygen -t rsa -b 2048 sudo cat /root/.ssh/id_rsa.pub ssh-rsa AAAAB3Nz4rFgt...vgaD root@node1 ``` 2. Get the IP address of the primary node (in the following example, `192.168.1.2`): ```shell ip addr 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:34:6c:35 brd ff:ff:ff:ff:ff:ff inet 192.168.1.2/24 brd 192.168.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe34:6c35/64 scope link valid_lft forever preferred_lft forever ``` 3. On each peer node, append the public key to `root`’s **authorized_keys** file. The `from=192.168.1.2` prefix restricts access to only the IP address of the primary node: ```shell sudo mkdir /root/.ssh sudo echo 'from="192.168.1.2" ssh-rsa AAAAB3Nz4rFgt...vgaD root@node1' >> /root/.ssh/authorized_keys ``` 4. Add the following line to **/etc/ssh/sshd_config**: ```none PermitRootLogin without-password ``` 5. Reload `sshd` on each peer (but not the primary) to allow SSH key authentication - for Amazon Linux, AlmaLinux, CentOS, Oracle Linux, RHEL, Rocky Linux: ```shell sudo systemctl restart sshd ``` - for Ubuntu, Debian, SLES: ```shell sudo systemctl restart ssh ``` 6. Verify that the `root` user can `ssh` to each of the other nodes without providing a password: ```shell sudo ssh root@node2.example.com ``` ### Creating the nginx-sync.conf Configuration File on the Primary Node On the primary node, create the file **/etc/nginx-sync.conf** with these contents: ```none NODES="node2.example.com node3.example.com node4.example.com" CONFPATHS="/etc/nginx/nginx.conf /etc/nginx/conf.d" EXCLUDE="default.conf" ``` #### Common Parameters Use a space or newline character to separate the items in each list: | Parameter | Description | | ------------------------ | -------------------------------------------------------------------------------------| | `NODES` | List of peers that receive the configuration from the primary. | | `CONFPATHS` | List of files and directories to distribute from the primary to the peers. | | `EXCLUDE` | (Optional) List of configuration files on the primary not to distribute to the peers.| #### Advanced Parameters | Parameter | Description | Default | | ------------------------ | ---------------------------------------------------------------------------------------|-------------------------| | `BACKUPDIR` | Location of backup on each peer | **/var/lib/nginx-sync** | | `DIFF` | Location of `diff` binary | **/usr/bin/diff** | | `LOCKFILE` | Location of the lock file used to ensure only one `nginx-sync` operation runs at a time| **/tmp/nginx-sync.lock**| | `NGINX` | Location of the **nginx-plus** binary | **/usr/sbin/nginx** | | `POSTSYNC` | Space-separated list of file substitutions to make on each remote node in the format:
`'\\|\'`
The substitution is applied in place:
`sed -i' ' \ \`
For example, to substitute the IP address of *node2.example.com* (*192.168.2.2*) for the IP address of *node1.example.com* (*192.168.2.1*) in *keepalived.conf*:
`POSTSYNC="/etc/keepalived/keepalived.conf\|'s/192\.168\.2\.1/192.168.2.2/'"` | | | `RSYNC` | Location of the `rsync` binary | **/usr/bin/rsync** | | `SSH` | Location of the `ssh` binary | **/usr/bin/ssh** | ### Testing the Configuration Back up the configuration before testing. - Synchronize configuration and reload F5 NGINX Plus on the peers – `nginx-sync.sh` - Display usage information – `nginx-sync.sh -h` - Compare configuration between the primary and a peer – `nginx-sync.sh -c ` - Compare configuration on the primary to all peers – `nginx-sync.sh -C` ## Frequently Asked Questions ### Why Do I Need to Grant SSH Access to `root`? The primary node needs to be able to remotely run commands on the peer as the `root` user (for example, `service nginx reload`), and needs to be able to update configuration files (for example, in **/etc/nginx/**) that are owned by `root`. It might seem that granting SSH access to `root` is giving away too many privileges, but it is important to remember that any process that can write remote NGINX Plus configuration and reload the remote NGINX Plus process can subvert this process to gain remote `root` access to the server. Therefore, assume that users who gain `root` access on the primary node also have `root` access on the peer nodes. ### How Do I Synchronize Configuration if the Primary Fails? If the primary fails and will not soon return to service, you need to promote a peer to operate as primary by following the instructions in [Installation](#instructions). This involves 1. [Installing the `nginx-sync.sh` script](#in_detail1) 2. [Granting SSH access to the remaining peers](#in_detail2) 3. [Creating the configuration file](#in_detail3) You can preconfigure several machines to operate as primary, but must ensure that only one node actually runs as primary at a given time. ### What Happens if a Peer Node Fails? If a peer node fails, it no longer receives configuration updates. The `nginx-sync.sh` script returns an error but continues to distribute the configuration to the remaining peers. When the node recovers, its configuration is out of date. You can display the configuration differences by running `nginx-sync.sh -c -d`: ```shell nginx-sync.sh -c node2.example.com -d ``` The output of the command: ```diff diff -ru /tmp/localconf.1XrIqP7f/etc/nginx/conf.d/responder.conf /tmp/remoteconf.Xq5LWGKU/etc/nginx/conf.d/responder.conf --- /tmp/localconf.1XrIqP7f/etc/nginx/conf.d/responder.conf 2020-09-25 10:29:36.988064021 -0800 +++ /tmp/remoteconf.Xq5LWGKU/etc/nginx/conf.d/responder.conf 2020-09-25 10:28:39.764066539 -0800 @@ -4,6 +4,6 @@ listen 80; location / { - return 200 "Received request on $server_addr on host $hostname blue\n"; + return 200 "Received request on $server_addr on host $hostname red\n"; } } * Synchronization ended at Fri Sep 25 18:30:49 UTC 2020 ``` The next time you run `nginx-sync.sh`, the node gets updated with the current primary configuration. -------------------------------------------------------------------------------- title: "Authentication Based on Subrequest Result" url: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/index.md description: Authenticate clients during request processing by making a subrequest to an external authentication service, such as LDAP or OAuth. -------------------------------------------------------------------------------- ## Introduction NGINX and F5 NGINX Plus can authenticate each request to your website with an external server or service. To perform authentication, NGINX makes an HTTP subrequest to an external server where it is verified. Such type of authentication allows implementing various authentication schemes, such as multifactor authentication, or allows implementing LDAP or OAuth authentication. ## Response Codes Subrequest Response Codes: - 2xx - access is allowed - 401, 403 - access is denied ## Prerequisites - NGINX Plus or NGINX Open Source - External authentication server or service ## Configuring NGINX and NGINX Plus 1. Make sure your NGINX Open Source is compiled with the `with-http_auth_request_module` configuration option. Run this command and verify that the output includes `--with-http_auth_request_module`: ```none nginx -V 2>&1 | grep -- 'http_auth_request_module' ``` Skip this step for NGINX Plus as it already includes the auth_request module. 2. In the location that requires request authentication, specify the [auth_request](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html#auth_request) directive and specify an internal location where an authorization subrequest will be forwarded to: ```nginx location /private/ { auth_request /auth; #... } ``` Here, for each request to **/private**, a subrequest to the internal **/auth** location will be made. 3. Specify an internal location and the [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive inside this location that will proxy authentication subrequests to an authentication server or service: ```nginx location = /auth { internal; proxy_pass http://auth-server; #... } ``` 4. As the request body is discarded for authentication subrequests, set the [proxy_pass_request_body](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_body) directive to `off` and also set the `Content-Length` header to a null string: ```nginx location = /auth { internal; proxy_pass http://auth-server; proxy_pass_request_body off; proxy_set_header Content-Length ""; #... } ``` 5. Pass the full original request URI with arguments with the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive: ```nginx location = /auth { internal; proxy_pass http://auth-server; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header X-Original-URI $request_uri; } ``` 6. As an option, you can set a variable value basing on the result of the subrequest with the [auth_request_set](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html#auth_request_set) directive: ```nginx location /private/ { auth_request /auth; auth_request_set $auth_status $upstream_status; } ``` ## Complete Example This example sums up the previous steps into one configuration: ```nginx http { #... server { #... location /private/ { auth_request /auth; auth_request_set $auth_status $upstream_status; } location = /auth { internal; proxy_pass http://auth-server; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header X-Original-URI $request_uri; } } } ``` -------------------------------------------------------------------------------- title: "Compression and Decompression" url: https://docs.nginx.com/nginx/admin-guide/web-server/compression/index.md description: Compress server responses, or decompress them for clients that don't support compression, to improve delivery speed and reduce overhead on the server. -------------------------------------------------------------------------------- This section describes how to configure compression or decompression of responses, as well as sending compressed files. ## Introduction Compressing responses often significantly reduces the size of transmitted data. However, since compression happens at runtime it can also add considerable processing overhead which can negatively affect performance. NGINX performs compression before sending responses to clients, but does not “double compress” responses that are already compressed (for example, by a proxied server). ## Enabling Compression To enable compression, include the [gzip](https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip) directive with the `on` parameter. ```nginx gzip on; ``` By default, NGINX compresses responses only with MIME type `text/html`. To compress responses with other MIME types, include the [gzip_types](https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_types) directive and list the additional types. ```nginx gzip_types text/plain application/xml; ``` To specify the minimum length of the response to compress, use the [gzip_min_length](https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_min_length) directive. The default is 20 bytes (here adjusted to 1000): ```nginx gzip_min_length 1000; ``` By default, NGINX does not compress responses to proxied requests (requests that come from the proxy server). The fact that a request comes from a proxy server is determined by the presence of the `Via` header field in the request. To configure compression of these responses, use the [gzip_proxied](https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_proxied) directive. The directive has a number of parameters specifying which kinds of proxied requests NGINX should compress. For example, it is reasonable to compress responses only to requests that will not be cached on the proxy server. For this purpose the `gzip_proxied` directive has parameters that instruct NGINX to check the `Cache-Control` header field in a response and compress the response if the value is `no-cache`, `no-store`, or `private`. In addition, you must include the `expired` parameter to check the value of the `Expires` header field. These parameters are set in the following example, along with the `auth` parameter, which checks for the presence of the `Authorization` header field (an authorized response is specific to the end user and is not typically cached): ```nginx gzip_proxied no-cache no-store private expired auth; ``` As with most other directives, the directives that configure compression can be included in the `http` context or in a `server` or `location` configuration block. The overall configuration of gzip compression might look like this. ```nginx server { gzip on; gzip_types text/plain application/xml; gzip_proxied no-cache no-store private expired auth; gzip_min_length 1000; ... } ``` ## Enabling Decompression Some clients do not support responses with the `gzip` encoding method. At the same time, it might be desirable to store compressed data, or compress responses on the fly and store them in the cache. To successfully serve both clients that do and do not accept compressed data, NGINX can decompress data on the fly when sending it to the latter type of client. To enable runtime decompression, use the [gunzip](https://nginx.org/en/docs/http/ngx_http_gunzip_module.html#gunzip) directive. ```nginx location /storage/ { gunzip on; ... } ``` The `gunzip` directive can be specified in the same context as the `gzip` directive: ```nginx server { gzip on; gzip_min_length 1000; gunzip on; ... } ``` Note that this directive is defined in a separate [module](https://nginx.org/en/docs/http/ngx_http_gunzip_module.html) that might not be included in an NGINX Open Source build by default. ## Sending Compressed Files To send a compressed version of a file to the client instead of the regular one, set the [gzip_static](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html#gzip_static) directive to `on` within the appropriate context. ```nginx location / { gzip_static on; } ``` In this case, to service a request for **/path/to/file**, NGINX tries to find and send the file **/path/to/file.gz**. If the file doesn’t exist, or the client does not support gzip, NGINX sends the uncompressed version of the file. Note that the `gzip_static` directive does not enable on-the-fly compression. It merely uses a file compressed beforehand by any compression tool. To compress content (and not only static content) at runtime, use the `gzip` directive. This directive is defined in a separate [module](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html) that might not be included in an NGINX Open Source build by default. -------------------------------------------------------------------------------- title: "Install NGINX Plus on the Google Cloud Platform" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus-google-cloud-platform/index.md description: Install F5 NGINX Plus on the Google Cloud Platform, to provide sophisticated Layer 7 load balancing for your apps running on Google Compute Engine. -------------------------------------------------------------------------------- [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus), the high‑performance application delivery platform, load balancer, and web server, is available on the Google Cloud Platform as a virtual machine (VM) image. The VM image contains the latest version of NGINX Plus, optimized for use with the Google Cloud Platform Compute Engine. ## Install the NGINX Plus VM To quickly set up an NGINX Plus environment on the Google Cloud Platform, perform the following steps. > **Note:** The Google Cloud Platform interface is under active development. Although we make every effort to provide accurate instructions, they are subject to change. Also, the exact options you see depend on whether or not you have existing projects. 1. Access the [NGINX Plus page](https://console.cloud.google.com/marketplace/details/nginx-public/nginx-plus) in Google Marketplace and click the **LAUNCH ON COMPUTE ENGINE** button. screenshot F5 NGINX Plus on Google Marketplace 2. The **Select or create a project** window opens. Enter a project name and select a value from the **Organization** drop‑down, then click the **Create** button. screenshot nginx plus to google computer engine 3. In the **Configure & Deploy** window, enter or select appropriate values for zone, machine type, and so on. Click the **Deploy** button. > **Note:** In the **Firewall** section, be sure the **Allow HTTP traffic** checkbox is checked. For more information on controlling incoming traffic, see the [Firewall Rules Overview](https://cloud.google.com/vpc/docs/firewalls) in the Google Cloud Platform documentation. Screenshot New NGINX Plus Deployment 4. Click the **Create** button. The Google Developers Console confirms that NGINX Plus was deployed. Screenshot NGINX plus deployed on Google Cloud Platform As soon as the project deploys and the new virtual machine (VM) instance starts running, NGINX Plus starts automatically and serves a default **index.html** page. To verify that NGINX Plus is working properly, use a web browser to access the public DNS name of the new VM and view the page. You can also check the status of the NGINX Plus server by logging into the VM and running this command: ```shell /etc/init.d/nginx status ``` ## What If I Need Help? If you encounter any problems with NGINX Plus configuration, documentation is available at [nginx.org](https://nginx.org/en/docs/) and in the [NGINX Plus Admin Guide](/nginx/admin-guide/installing-nginx/). Customers who purchase an NGINX Plus VM image on the Google Cloud Platform are eligible for the Google Cloud Platform VM support provided by the NGINX, Inc. engineering team. To activate support, submit the [Google Cloud Platform Support Activation](https://www.nginx.com/gcp-support-activation/) form. ### Access the Open Source Licenses for NGINX Plus NGINX Plus includes open source software written by NGINX, Inc. and other contributors. The text of the open source licenses is provided in Appendix B of the _NGINX Plus Reference Guide_. To access the guide included with the NGINX Plus VM instance, run this command: ```shell less /usr/share/nginx/html/nginx-modules-reference.pdf ``` The _NGINX Plus Reference Guide_ is also [available online](http://www.nginx.com/wp-content/uploads/2023/08/nginx-modules-reference.pdf). -------------------------------------------------------------------------------- title: "NGINX Diagnostic Package" url: https://docs.nginx.com/nginx/admin-guide/monitoring/diagnostic-package/index.md description: This page describes how to trigger the automatic collection of data required to troubleshoot issues in a NGINX or F5 NGINX Plus deployment. -------------------------------------------------------------------------------- ## Overview NGINX Diagnostic Package is used to obtain additional information needed by [F5 Technical Support](https://account.f5.com/myf5) when troubleshooting your issue. The package is created by a script that can be [downloaded](https://nginx.org/download/nginx-supportpkg.sh) from the [nginx.org](https://nginx.org/download/) website. The script collects the following information: - host commands such as `ps`, `lsof`, `vmstat` - NGINX configuration files - NGINX log files - NGINX service information - NGINX process information - NGINX versions, dynamically linked libraries - NGINX Plus API endpoints - NGINX Agent logs and configs if NGINX Agent is present - F5 WAF for NGINX logs and configs if F5 WAF for NGINX is present The script does not collect or create: - njs scripts - Lua scripts - Core dumps **Note:** It is highly recommended that you review the script and the created resources and verify that they conform with your organization's data sharing policies. ## Supported Operating Systems The script can be run on most [operating systems supported by NGINX](https://docs.nginx.com/nginx/technical-specs/) and has been tested on the following operating systems: - AlmaLinux 9.1 - Amazon Linux 2 - CentOS 7 - Debian 11 - RHEL 9.1 - Rocky Linux 9.1 - SUSE Linux Enterprise Server 15 - Ubuntu 20.04 ## Usage To create NGINX Diagnostic Package: 1. [Download](https://nginx.org/download/nginx-supportpkg.sh) the `nginx-supportpkg.sh` script: ```shell wget https://nginx.org/download/nginx-supportpkg.sh ``` 2. Grant execution permissions to the script: ```shell chmod +x nginx-supportpkg.sh ``` 3. Run the script. The script requires root privileges to run. The script can be run with optional arguments, see [Arguments](#arguments) for details. ```shell sudo ./nginx-supportpkg.sh ``` The created package will be located in the same directory as the current script. It is a `.tar.gz` archive named according to the file name pattern: `support-pkg-.tar.gz`. 4. After the package has been created, it is recommended to extract and review its contents. Use the `tar` command to extract the archive: ```shell tar -xvf support-pkg-1682457903.tar.gz ``` The archive contains textual output of all the commands run by the script to make it easier to review the collected data. ## Arguments The following table lists the arguments you can use to customize the data that needs to be collected based on your NGINX deployment. | Short | Long | Description | Example | Default | | ----- | ------------------------ | ----------------------------------------------------------------------| -------------------------| -----------------| | `-h` | `--help` | Prints information about the script arguments to stdout. | `--help` | N/A | | `-d` | `--debug` | Sets bash debug flag. | `--debug` | N/A | | `-o` | `--output_dir` | The output directory where the tar archive is saved. | `-o ~/output` | `$(pwd)` | | `-n` | `--nginx_log_path` | The directory where the NGINX log files are located. | `-n /var/log/nginx` | `/var/log/nginx` | | `-xc` | `--exclude_nginx_configs`| Excludes all NGINX configuration files from the support package. | `--exclude_nginx_configs`| N/A | | `-xl` | `--exclude_nginx_logs` | Excludes all NGINX log files from the support package. | `--exclude_nginx_logs` | N/A | | `-ac` | `--exclude_agent_configs`| Excludes all NGINX Agent configuration files from the support package.| `--exclude_agent_configs`| N/A | | `-al` | `--exclude_agent_logs` | Excludes all NGINX Agent logs from the support package. | `--exclude_agent_logs` | N/A | | `-nc` | `--exclude_nap_configs` | Excludes all F5 WAF for NGINX config files from the support package. | `--exclude_nap_configs` | N/A | | `-nl` | `--exclude_nap_logs` | Excludes all F5 WAF for NGINX log files from the support package. | `--exclude_nap_logs` | N/A | | `-ea` | `--exclude_api_stats` | Excludes NGINX Plus API stats from the support package. | `--exclude_api_stats` | N/A | | `-pi` | `--profile_interval` | Profiling interval in seconds. | `-pi 20` | 15 | -------------------------------------------------------------------------------- title: "Runtime State Sharing in a Cluster" url: https://docs.nginx.com/nginx/admin-guide/high-availability/zone_sync/index.md description: This article describes how to use F5 NGINX Plus to synchronize shared memory zones across NGINX cluster nodes including sticky learn session persistence, requests limiting, and key-value store data. -------------------------------------------------------------------------------- ## Introduction If several F5 NGINX Plus instances are organized in a cluster, they can share some state data between them, including: - [sticky learn](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#sticky) session persistence - [requests limiting](https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/#limit_req) - [key-value storage](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval_zone) All NGINX Plus instances can exchange state data with all other members in a cluster, provided that the shared memory zone has the same name on all cluster members. ## Prerequisites - NGINX Plus [R16](nginx/releases.md#r16) for sharing limits of requests processing and key-value data across the cluster - NGINX Plus [R15](nginx/releases.md#r15) for sharing limits of sticky learn data across the cluster State sharing across a cluster is eventually consistent by nature. It is strongly recommended using data-center grade networks for clustering traffic, as latency, low bandwidth, and packet loss will have a significant negative impact on state consistency. We do not recommend stretching clusters over the Internet, regions, or availability zones. ## Configuring zone synchronization For each NGINX instance in a cluster, open the NGINX configuration file and perform the following steps: 1. Enable synchronization between cluster nodes: in the top-level [`stream`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) block, create a [`server`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#server) with the [`zone_sync`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync) directive: ```nginx stream { #... server { zone_sync; #... } } ``` 2. Specify all other NGINX instances in a cluster with the [`zone_sync_server`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_server) directive. Cluster nodes can be added dynamically using the DNS service if the [`resolver`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#resolver) is used: ```nginx stream { resolver 10.0.0.53 valid=20s; server { zone_sync; zone_sync_server nginx-cluster.example.com:9000 resolve; } } ``` Otherwise, each cluster node can be added statically as a separate line of the [`zone_sync_server`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_server) directive: ```nginx stream { server { zone_sync; zone_sync_server nginx-node1.example.com:9000; zone_sync_server nginx-node2.example.com:9000; zone_sync_server nginx-node3.example.com:9000; } } ``` 3. Enable SSL by specifying the `ssl` parameter of the [`listen`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#listen) directive for the TCP server: ```nginx stream { resolver 10.0.0.53 valid=20s; server { listen 10.0.0.1:9000 ssl; #... zone_sync; zone_sync_server nginx-cluster.example.com:9000 resolve; #... } } ``` 4. Specify the path to the certificates with the [`ssl_certificate`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_certificate) directive, and to the private key with the [`ssl_certificate_key`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_certificate_key) directive. Both the certificate and key must be in the PEM format: ```nginx stream { resolver 10.0.0.53 valid=20s; server { listen 10.0.0.1:9000 ssl; ssl_certificate /etc/ssl/nginx-1.example.com.server_cert.pem; ssl_certificate_key /etc/ssl/nginx-1.example.com.key.pem; zone_sync; zone_sync_server nginx-cluster.example.com:9000 resolve; #... } } ``` 5. Enable SSL connections between cluster servers with the [`zone_sync_ssl`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl) directive, and enable verification of another cluster server certificate with [`zone_sync_ssl_verify`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl_verify) and [`zone_sync_ssl_trusted_certificate`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl_trusted_certificate) directives: ```nginx stream { resolver 10.0.0.53 valid=20s; server { listen 10.0.0.1:9000 ssl; ssl_certificate /etc/ssl/nginx-1.example.com.server_cert.pem; ssl_certificate_key /etc/ssl/nginx-1.example.com.key.pem; zone_sync; zone_sync_server nginx-cluster.example.com:9000 resolve; zone_sync_ssl on; zone_sync_ssl_verify on; zone_sync_ssl_trusted_certificate /etc/ssl/server_ca.pem; #... } } ``` 6. Set up certificate-based authentication across cluster nodes. Add the [`zone_sync_ssl_certificate`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl_certificate) and [`zone_sync_ssl_certificate_key`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl_certificate_key`) directives to send the client certificate for outgoing connections. Then configure NGINX to require client certificates by setting the [`ssl_verify_client`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_verify_client) directive to `on` and specifying the location of your client certificates CA with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html#ssl_trusted_certificate) directive: ```nginx stream { resolver 10.0.0.53 valid=20s; server { listen 10.0.0.1:9000 ssl; ssl_certificate /etc/ssl/nginx-1.example.com.server_cert.pem; ssl_certificate_key /etc/ssl/nginx-1.example.com.key.pem; zone_sync; zone_sync_server nginx-cluster.example.com:9000 resolve; zone_sync_ssl on; zone_sync_ssl_verify on; zone_sync_ssl_trusted_certificate /etc/ssl/server_ca.pem; zone_sync_ssl_certificate localhost.crt; zone_sync_ssl_certificate_key localhost.key; ssl_verify_client on; ssl_trusted_certificate /etc/ssl/client_ca.pem; #... } } ``` ### Fine-tuning Synchronization Generally you do not have to tune sync options, but in some cases it can be useful to adjust some of these values: ```nginx #... zone_sync; zone_sync_server nginx-cluster.example.com:9000 resolve; zone_sync_buffers 256 4k; zone_sync_connect_retry_interval 1s; zone_sync_connect_timeout 5s; zone_sync_interval 1s; zone_sync_timeout 5s; #... ``` [`zone_sync_buffers`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_buffers) - controls the number of buffers and their size. Increasing the number of buffers will increase the number of information stored in them. [`zone_sync_connect_retry_interval`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_connect_retry_interval) - sets the timeout between connection attempts to a cluster node. [`zone_sync_connect_timeout`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_connect_timeout) - sets the time required to connect to a cluster node. [`zone_sync_interval`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_interval) - sets an interval for polling updates in a shared memory zone. Increasing this value may result in data inconsistency between cluster nodes, decreasing the value may lead to high consumption of cpu and memory resources. [`zone_sync_timeout`](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_timeout) - sets the lifetime of a TCP stream between cluster nodes. If a TCP stream is idle for longer than the value, the connection will be closed. ## Managing Cluster Nodes ### Starting a Node To start a new node: - in case of DNS, update a DNS record of a cluster hostname with the IP address of the new node and start an instance - in case of statically added nodes, add the node's address to nginx configuration file and reload all other nodes When the node is started, it discovers other nodes from DNS or static configuration and starts sending updates. Other nodes eventually discover the new node using DNS and start pushing updates to it. ### Stopping a Node To stop a node, send the 'QUIT' signal: ```shell nginx -s quit ``` As soon as the node receives the signal, it finishes zone synchronization and gracefully closes open connections. ### Removing a Node To remove a node: - in case of DNS, update a DNS record of a cluster hostname and remove the node's IP address - in case of statically added nodes, remove the node's address from nginx configuration file on each node and reload each node. When the node is removed, other nodes close connections to the removed node and will no longer try to connect to it. After the node is removed, it can be stopped. ## Using synchronization in a cluster ### Sticky learn zone synchronization If your existing configuration already uses the [sticky learn](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#sticky) feature, existing state can be simply shared across a cluster by adding the [`sync`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_learn_sync) parameter to the existing `sticky` directive in the configuration file of each NGINX instance in a cluster. Note that the zone name must be the same in all other NGINX nodes in the cluster: ```nginx upstream my_backend { zone my_backend 64k; server backends.example.com resolve; sticky learn zone=sessions:1m create=$upstream_cookie_session lookup=$cookie_session sync; } server { listen 80; location / { proxy_pass http://my_backend; } } ``` See [Enabling Session Persistence](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#sticky) for information how to configure the "sticky learn" session persistence method. ### Request limits zone synchronization If your existing configuration already uses rate limiting, these limits can be applied across a cluster by simply adding the [`sync`](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) parameter to the [`limit_req_zone`](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive in the configuration file of each NGINX instance in a cluster: ```nginx limit_req_zone $remote_addr zone=req:1M rate=100r/s sync; server { listen 80; location / { limit_req zone=req; proxy_pass http://my_backend; } } ``` The zone name also must be the same in all other NGINX nodes in the cluster. See [Limiting the Request Rate](https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/#limit_req) for more information. ### Key-value storage zone synchronization Similar to rate limiting and sticky learn, the contents of the key-value shared memory zone can be shared across NGINX machines in a cluster with the `sync` parameter of the [`keyval_zone`](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval_zone) directive: ```nginx keyval_zone zone=one:32k state=/var/lib/nginx/state/one.keyval sync; keyval $arg_text $text zone=one; #... server { #... location / { return 200 $text; } location /api { api write=on; } } ``` See [Dynamic Denylisting of IP Addresses](/nginx/admin-guide/security-controls/denylisting-ip-addresses.md) for information how to configure and manage the key-value storage. ## Monitoring Cluster State Cluster state data can be monitored with [NGINX Plus API metrics](https://nginx.org/en/docs/http/ngx_http_api_module.html#stream_zone_sync_): - names of shared memory zones - total number of records on node - number of records that needs to be sent - sync status per each node in the cluster ## Configuring the API In order to get access to API metrics, you will need to configure the API: 1. Enable the NGINX Plus API in read‑write mode with the [`api`](https://nginx.org/en/docs/http/ngx_http_api_module.html#api) directive: ```nginx # ... server { listen 80; server_name www.example.com; location /api { api write=on; } } ``` 2. It is highly recommended to [restrict access](/nginx/admin-guide/security-controls/controlling-access-proxied-http.md) to this location, for example by allowing access only from localhost (`127.0.0.1`), and by restricting access to `PATCH`, `POST`, and `DELETE` methods to some users with HTTP basic authentication: ```nginx # ... server { listen 80; server_name www.example.com; location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } api write=on; allow 127.0.0.1; deny all; } } ``` See [Using the API for Dynamic Configuration](https://docs.nginx.com/nginx/admin-guide/load-balancer/dynamic-configuration-api/#api_use) for instructions how to configure and use NGINX Plus API. ## Polling Sync Status with the API To get the synchronization status of the shared memory zone, send the API command, for example, with `curl`: ```shell curl -s '127.0.0.1/api/9/stream/zone_sync' | jq ``` The output will be: ```json { "zones" : { "zone1" : { "records_pending" : 2061, "records_total" : 260575 }, "zone2" : { "records_pending" : 0, "records_total" : 14749 } }, "status" : { "bytes_in" : 1364923761, "msgs_in" : 337236, "msgs_out" : 346717, "bytes_out" : 1402765472, "nodes_online" : 15 } } ``` If all nodes have approximately the same number of records ([`records_total`](https://nginx.org/en/docs/http/ngx_http_api_module.html#def_nginx_stream_zone_sync_zone)) and almost empty outgoing queue ([`records_pending`](https://nginx.org/en/docs/http/ngx_http_api_module.html#def_nginx_stream_zone_sync_zone)), the cluster may be considered healthy. -------------------------------------------------------------------------------- title: "Single Sign-On with Microsoft Entra ID" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/entra-id/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Microsoft Entra ID (formerly Azure Active Directory) as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Microsoft Entra ID](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id) as the Identity Provider (IdP), and NGINX Plus as the Relying Party, or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. In the latest NGINX Plus version, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - A Microsoft Entra tenant with admin access. - Azure CLI. For installation instructions, see [How to install the Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure Entra ID {#entra-setup} Register a new application in Microsoft Entra ID that will represent NGINX Plus as an OIDC client. This is necessary to obtain unique identifiers and secrets for OIDC, as well as to specify where Azure should return tokens. Ensure you have access to the Azure Portal with Entra ID app administrator privileges. ### Register new Azure Web Application 1. Log in to Azure CLI: ```shell az login ``` This command will open your default browser for authentication. 2. Register a New Application. - Create a new application, for example, "Nginx Demo App", with NGINX callback URI `/oidc_callback`: ```shell az ad app create --display-name "Nginx Demo App" --web-redirect-uris "https://demo.example.com/oidc_callback" ``` - From the command output, copy the `appId` value which represents your **Client ID**. You will need it later when configuring NGINX Plus. 3. Generate a new Client Secret. - Create a client secret for your application by running: ```bash az ad app credential reset --id ``` - Replace the `` with the value obtained in the previous step. - From the command output, copy the the `password` value which represents your **Client Secret**. You will need it later when configuring NGINX Plus. Make sure to securely save the generated client secret, as it will not be displayed again. - From the same command output, copy the the `tenant` value which represents your **Tenant ID**. You will need it later when configuring NGINX Plus. 4. Configure logout URLs to support RP-initiated and front-channel logout: - Add a logout URL for your application by running: ```bash az ad app update --id --web-logout-urls "https://demo.example.com/post_logout/" ``` - Replace the `` with the value obtained in step 2. - To enable OpenID Connect front-channel logout (single sign-out when the user signs out of another application), configure the *Front-channel logout URL* for your application in the Microsoft Entra admin center. Set it to the absolute HTTPS URL that matches the `frontchannel_logout_uri` you will configure in NGINX Plus, for example `https://demo.example.com/front_logout`. According to the OpenID Connect front-channel logout specification, the Identity Provider sends both the issuer (`iss`) and the session identifier (`sid`) as query parameters. Microsoft Entra ID currently sends only the `sid` value; the NGINX Plus OIDC module for NGINX Plus Release 36 supports both the fully compliant (`iss` + `sid`) and the Entra-specific (`sid`‑only) variants and will clear the corresponding user session in either case. ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, Microsoft Entra ID publishes the `.well-known/openid-configuration` document at the following address: `https://login.microsoftonline.com//v2.0/.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https://login.microsoftonline.com//v2.0/.well-known/openid-configuration | jq ``` Where: - the `` is your Microsoft Entra Tenant ID - the `login.microsoftonline.com` is your Microsoft Entra server address - the `/v2.0/.well-known/openid-configuration` is the default address for Microsoft Entra ID for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://login.microsoftonline.com/{tenant_id}/v2.0", "authorization_endpoint": "https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize", "token_endpoint": "https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token", "jwks_uri": "https://login.microsoftonline.com/{tenant_id}/discovery/v2.0/keys", "userinfo_endpoint": "https://graph.microsoft.com/oidc/userinfo", "end_session_endpoint": "https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/logout", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for Microsoft Entra ID is `https://login.microsoftonline.com//v2.0`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Tenant ID** in the next steps. ## Set up NGINX Plus {#nginx-plus} With Microsoft Entra ID configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.0 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Tenant ID** obtained during [Microsoft Entra ID Configuration](#entra-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the Entra ID provider named `entra` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider entra { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - your **Client ID** obtained in [Entra ID Configuration](#entra-setup) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - your **Client Secret** obtained in [Entra ID Configuration](#entra-setup) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - the **Issuer** URL obtained in [Entra ID Configuration](#entra-setup) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically: `https://login.microsoftonline.com//v2.0`. By default, NGINX Plus creates the metadata URL by appending the `/.well-known/openid-configuration` part to the Issuer URL. If your metadata URL is different, you can explicitly specify it with the [`config_url`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url) directive. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where Microsoft Entra ID should redirect the user after a successful logout. This value **must also be configured** in the Entra ID application logout URLs. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user's ID token as a *hint* to Microsoft Entra ID. This directive is **optional**, however, if it is omitted the Microsoft Entra ID may display an extra confirmation page asking the user to approve the logout request. If the “Require ID token in logout requests” option is enabled in your tenant (commonly the case in Azure AD B2C), then the token hint becomes **mandatory**. - The **frontchannel_logout_uri** directive defines the URI that receives OpenID Connect front-channel logout requests from Microsoft Entra ID. This URI must be an HTTPS path hosted by NGINX Plus and must match the *Front-channel logout URL* configured in the Entra ID application registration. When a front-channel logout GET request is received at this URI (typically in a hidden iframe), the OIDC module clears the local session for the affected user. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch userinfo from Microsoft Graph API and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when the provider metadata advertises the `S256` code challenge method in the `code_challenge_methods_supported` field of the discovery document. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP metadata does not list `S256`. - The module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`), which remains the default for Microsoft Entra ID. - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider entra { issuer https://login.microsoftonline.com//v2.0; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; frontchannel_logout_uri /front_logout; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with Entra ID OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `entra` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#entra-setup-oidc-provider): ```nginx # ... location / { auth_oidc entra; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by Entra ID: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by Entra ID - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc entra; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nEntra ID sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider entra { # The issuer is typically something like: # https://login.microsoftonline.com//v2.0 issuer https://login.microsoftonline.com//v2.0; # Replace with your actual Entra client_id and client_secret client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Front-channel logout (OP‑initiated single sign-out) frontchannel_logout_uri /front_logout; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration (enabled automatically when supported by the IdP) # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Protect this location with Entra OIDC auth_oidc entra; # Forward OIDC claims as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nEntra ID sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open `https://demo.example.com/` in a browser. You will be automatically redirected to the Entra ID sign-in page. 2. Enter valid Entra ID credentials of a user who has access the application. Upon successful sign-in, Entra ID redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; Microsoft Entra ID ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to Microsoft Entra ID for a fresh sign‑in, proving the session has been terminated. **Note:** If you restricted access to a group of users, be sure to select a user who has access to the application. ## See Also - [Microsoft identity platform documentation](https://learn.microsoft.com/en-us/entra/identity-platform/) - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added front-channel logout support (`frontchannel_logout_uri`), PKCE configuration (`pkce` directive), and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "TCP Health Checks" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-health-check/index.md description: Monitor the health of TCP servers in an upstream group by sending periodic health checks, including customizable active health checks in F5 NGINX Plus. -------------------------------------------------------------------------------- ## Introduction NGINX and F5 NGINX Plus can continually test your TCP upstream servers, avoid the servers that have failed, and gracefully add the recovered servers into the load‑balanced group. ## Prerequisites - You have configured an upstream group of TCP servers in the [`stream`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context, for example: ```nginx stream { #... upstream stream_backend { server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12345; } #... } ``` - You have configured a server that passes TCP connections to the server group: ```nginx stream { #... server { listen 12345; proxy_pass stream_backend; } #... } ``` ## Passive TCP Health Checks If an attempt to connect to an upstream server times out or results in an error, NGINX Open Source or NGINX Plus can mark the server as unavailable and stop sending requests to it for a defined amount of time. To define the conditions under which NGINX considers an upstream server unavailable, include the following parameters to the [`server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) directive - [`fail_timeout`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#fail_timeout) – The amount of time within which a specified number of connection attempts must fail for the server to be considered unavailable. Also, the amount of time that NGINX considers the server unavailable after marking it so. - [`max_fails`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_fails) – The number of failed attempts that happen during the specified time for NGINX to consider the server unavailable. The default values are `10` seconds and `1` attempt. So if a connection attempt times out or fails at least once in a 10‑second period, NGINX marks the server as unavailable for 10 seconds. The example shows how to set these parameters to 2 failures within 30 seconds: ```nginx upstream stream_backend { server backend1.example.com:12345 weight=5; server backend2.example.com:12345 max_fails=2 fail_timeout=30s; server backend3.example.com:12346 max_conns=3; } ``` ### Server Slow Start A recently recovered upstream server can be easily overwhelmed by connections, which may cause the server to be marked as unavailable again. Slow start allows an upstream server to gradually recover its weight from zero to its nominal value after it has been recovered or became available. This can be done with the [`slow_start`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#slow_start) parameter of the upstream [`server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) directive: ```nginx upstream backend { server backend1.example.com:12345 slow_start=30s; server backend2.example.com; server 192.0.0.1 backup; } ``` Note that if there is only a single server in a group, the `slow_start` parameter is ignored and the server is never marked unavailable. Slow start is exclusive to NGINX Plus. ## Active TCP Health Checks Health checks can be configured to test a wide range of failure types. For example, NGINX Plus can continually test upstream servers for responsiveness and avoid servers that have failed. NGINX Plus sends special health check requests to each upstream server and checks for a response that satisfies certain conditions. If a connection to the server cannot be established, the health check fails, and the server is considered unhealthy. NGINX Plus does not proxy client connections to unhealthy servers. If several health checks are configured for an upstream group, the failure of any check is enough to consider the corresponding server unhealthy. To enable active health checks: 1. Specify a _shared memory zone_ – a special area where the NGINX Plus worker processes share state information about counters and connections. Add the [`zone`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#zone) directive to the upstream server group and specify the zone name (here, **stream_backend**) and the amount of memory (64 KB): ```nginx stream { #... upstream stream_backend { zone stream_backend 64k; server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12345; } #... } ``` 2. Enable active health checks for the upstream group with the [`health_check`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive: ```nginx stream { #... server { listen 12345; proxy_pass stream_backend; health_check; #... } } ``` 3. If necessary, reduce a timeout between two consecutive health checks with the [`health_check_timeout`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_timeout) directive. This directive overrides the [`proxy_timeout`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) value for health checks, as for health checks this timeout needs to be significantly shorter: ```nginx stream { #... server { listen 12345; proxy_pass stream_backend; health_check; health_check_timeout 5s; } } ``` 4. By default, NGINX Plus sends health check messages to the port specified by the [`server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) directive in the [`upstream`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) block. You can specify another port for health checks, which is particularly helpful when monitoring the health of many services on the same host. To override the port, specify the [`port`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_port) parameter of the [`health_check`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive: ```nginx stream { #... server { listen 12345; proxy_pass stream_backend; health_check port=12346; health_check_timeout 5s; } } ``` ### Fine-Tuning TCP Health Checks By default, NGINX Plus tries to connect to each server in an upstream server group every `5` seconds. If the connection cannot be established, NGINX Plus considers the health check failed, marks the server as unhealthy, and stops forwarding client connections to the server. To change the default behavior, include parameters to the [`health_check`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive: - [`interval`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_interval) – How often (in seconds) NGINX Plus sends health check requests (default is `5` seconds) - [`passes`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_passes) – Number of consecutive health checks the server must respond to to be considered healthy (default is `1`) - [`fails`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_fails) – Number of consecutive health checks the server must fail to respond to to be considered unhealthy (default is `1`) ```nginx stream { #... server { listen 12345; proxy_pass stream_backend; health_check interval=10 passes=2 fails=3; } #... } ``` In the example, the time between TCP health checks is increased to `10` seconds, the server is considered unhealthy after `3` consecutive failed health checks, and the server needs to pass `2` consecutive checks to be considered healthy again. ### The “match {}” Configuration Block You can create your own tests to verify server responses to health checks. These tests are defined with the [`match {}`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) configuration block placed in the [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context. 1. On the [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) level, specify the [`match {}`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) block and name it, for example, `tcp_test`: ```nginx stream { #... match tcp_test { #... } } ``` This block will contain tests described in **Step 3**. 2. Refer to the block from the [`health_check`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive by specifying the [`match`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_match) parameter and the name of the [`match`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) block: ```nginx stream { #... server { listen 12345; health_check match=tcp_test; proxy_pass stream_backend; } #... } ``` 3. Within the [`match`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) block, specify the conditions or tests under which a health check succeed. The block can accept the following parameters: - [`send`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match_send) – The text string or hexadecimal literals (“\x” followed by two hex digits) to send to the server - [`expect`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match_expect) – Literal string or regular expression that the data returned by the server needs to match These parameters can be used in different combinations, but no more than one `send` and one `expect` parameter can be specified at a time: - If no `send` or `expect` parameters are specified, the ability to connect to the server is tested. - If the `expect` parameter is specified, the server is expected to unconditionally send data first: ```nginx match pop3 { expect ~* "\+OK"; } ``` - If the `send` parameter is specified, it is expected that the connection will be successfully established and the specified string will be sent to the server: ```nginx match pop_quit { send QUIT; } ``` - If both the `send` and `expect` parameters are specified, then the string from the `send` parameter must match the regular expression from the `expect` parameter: ```nginx stream { #... upstream stream_backend { zone upstream_backend 64k; server backend1.example.com:12345; } match http { send "GET / HTTP/1.0\r\nHost: localhost\r\n\r\n"; expect ~* "200 OK"; } server { listen 12345; health_check match=http; proxy_pass stream_backend; } } ``` The example shows that in order for a health check to pass, the HTTP request must be sent to the server, and the expected result from the server contains `200` `OK` to indicate a successful HTTP response. -------------------------------------------------------------------------------- title: "Technical Specs" url: https://docs.nginx.com/nginx/technical-specs/index.md description: Platforms supported by F5 NGINX Plus and dynamically loaded modules, supported SSL/TLS versions, supported deployment environments, and list of modules built into NGINX Plus. -------------------------------------------------------------------------------- NGINX Plus is available only as a binary; it is not distributed as source code. For additional platforms and modules, [contact us](https://www.f5.com/products/get-f5). ## Supported Distributions {#supported-distributions} | Distribution | Supported on [PLS.37.1](nginx/releases.md#r37.0) | Supported on [PLS.37.0](nginx/releases.md#r36) | |------------------------------|---------------------------------------------------------------------------------|---------------------------------------------------------| | AlmaLinux | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64)
10 (x86_64, aarch64) | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64)
10 (x86_64, aarch64) | | Alpine Linux | 3.21 (x86_64, aarch64)
3.22 (x86_64, aarch64)
3.23 (x86_64, aarch64)
3.24 (x86_64, aarch64) | 3.21 (x86_64, aarch64)
3.22 (x86_64, aarch64)
3.23 (x86_64, aarch64) | | Amazon Linux | 2023 (x86_64, aarch64) | 2 (x86_64, aarch64)
2023 (x86_64, aarch64) | | Debian | 11 (x86_64, aarch64)
12 (x86_64, aarch64)
13 (x86_64, aarch64) | 11 (x86_64, aarch64)
12 (x86_64, aarch64)
13 (x86_64, aarch64) | | FreeBSD | 14.3+ (amd64)
15.0+ (amd64) | 13.5+ (amd64)
14.3+ (amd64)
15.0+ (amd64) | | Oracle Linux | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64) | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64) | | Red Hat Enterprise Linux | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64)
10.1+ (x86_64, aarch64) | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64)
10.1+ (x86_64, aarch64) | | Rocky Linux | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64)
10.1+ (x86_64, aarch64) | 8.1+ (x86_64, aarch64)
9.7+ (x86_64, aarch64)
10.1+ (x86_64, aarch64) | | SUSE Linux Enterprise Server | 15 SP7+ (x86_64)
16 (x86_64, aarch64) | 15 SP7+ (x86_64)
16 (x86_64, aarch64) | | Ubuntu | 22.04 (x86_64, aarch64)
24.04 (x86_64, aarch64)
26.04 (x86_64, aarch64) | 22.04 (x86_64, aarch64)
24.04 (x86_64, aarch64)
26.04 (x86_64, aarch64) | ## Dynamic Modules [Dynamic modules](/nginx/admin-guide/dynamic-modules/dynamic-modules.md#nginx-authored-and-nginx-certified-dynamic-modules) are supported on the [same distributions as NGINX Plus](#supported-distributions), unless noted otherwise in the table below. | | [F5 WAF for NGINX](/waf/changelog.md#packages) | [GeoIP](/nginx/admin-guide/dynamic-modules/geoip.md) | [HA-Keepalived](/nginx/admin-guide/high-availability/ha-keepalived-nodes.md#configuring-keepalived-for-an-additional-passive-node) | [NGINX sync](/nginx/admin-guide/high-availability/configuration-sharing.md#installing-nginx-sync-on-the-primary-machine) | |------------------------|----------------------|------------------|-----------------------------|---------------| | **RHEL-based*** | See [Packages](/waf/changelog/_index.md#packages) | ❌ | ✓ | ✓ | | **Alpine Linux** | See [Packages](/waf/changelog/_index.md#packages) | ✓ | ❌ | ❌ | | **Amazon Linux 2023** | See [Packages](/waf/changelog/_index.md#packages) | ❌ | ❌ | ✓ | | **Debian**, **Ubuntu** | See [Packages](/waf/changelog/_index.md#packages) | ✓ | ✓ | ✓ | | **FreeBSD** | ❌ | ❌ | ❌ | ❌ | | **SLES** | ❌ | 15 only| ✓ | ✓ | *RHEL-based operating systems include Red Hat Enterprise Linux, AlmaLinux, Oracle Linux, Rocky Linux. ## Supported SSL/TLS versions NGINX Plus supports the following SSL/TLS protocols: - SSLv2 - SSLv3 - TLSv1 - TLSv1.1 - TLSv1.2 - TLSv1.3 You can configure which protocols to enable or disable with the [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) directive. TLSv1.2 and earlier are supported on all operating systems listed in [Supported Distributions](#supported-distributions). TLSv1.3 is supported starting from NGINX Plus R17 and is enabled by default in NGINX Plus R29 and later. It requires OpenSSL 1.1.1 or higher. Note that not all operating systems supported by NGINX Plus include OpenSSL 1.1.1. Check your operating system's documentation to confirm TLSv1.3 compatibility. TLSv1.2 and TLSv1.3 are the default SSL protocols starting from NGINX Plus R34 (if supported by the OpenSSL library). If OpenSSL 1.0.0 or older is used, the default SSL protocols are TLSv1 and TLSv1.1. **Note:** NGINX Plus is built on the latest minor release of each supported operating system platform. In many cases, the latest revisions of these operating systems are adapting their platforms to support OpenSSL 3.5 (for example, RHEL 9.7 and 10.1). In these situations, NGINX Plus requires that OpenSSL 3.5.0 or later is installed for proper operation. ## Supported Deployment Environments - Bare metal - [Containers](/nginx/admin-guide/installing-nginx/installing-nginx-docker.md) and Kubernetes - Public clouds: AWS, Google Cloud Platform, Microsoft Azure - Virtual machines ## Recommended Hardware See [Sizing Guide for Deploying NGINX Plus](https://www.f5.com/pdf/deployment-guide/Sizing-Guide-for-Deploying-NGINX-Plus-in-Virtualized-Environments-2021-06-03.pdf) ## Modules in the NGINX Plus Package ### Core - [Core](https://nginx.org/en/docs/ngx_core_module.html) – Control basic functioning (mutexes, events, thread pools, workers, and so on) ### Clustering - [Zone Sync](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) – Synchronize [shared memory zones](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#zone) among nodes in a cluster ### HTTP Core - [HTTP Core](https://nginx.org/en/docs/http/ngx_http_core_module.html) – Process HTTP traffic - [Addition](https://nginx.org/en/docs/http/ngx_http_addition_module.html) – Prepend and append data to a response - [Auto Index](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html) – Generate directory listings - [Charset](https://nginx.org/en/docs/http/ngx_http_charset_module.html) – Add character set in `Content-Type` field of HTTP response header, and define or convert between character sets - [Empty GIF](https://nginx.org/en/docs/http/ngx_http_empty_gif_module.html) – Generate empty image response - [Gunzip](https://nginx.org/en/docs/http/ngx_http_gunzip_module.html) – Decompress responses for clients that don’t support compression - [Gzip](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) – Use GZIP to compress HTTP responses - [Gzip Static](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html) – Serve pre-compressed files from disk - [Headers](https://nginx.org/en/docs/http/ngx_http_headers_module.html) – Add fields to HTTP response headers, including `Cache-Control` and `Expires` - [Index](https://nginx.org/en/docs/http/ngx_http_index_module.html) – Specify index files used in directory requests - [Internal Redirect](https://nginx.org/en/docs/http/ngx_http_internal_redirect_module.html) – Allow internal redirects after checking request or connection processing limits, and access limits - [Random Index](https://nginx.org/en/docs/http/ngx_http_random_index_module.html) – Select random index file for directory request - [Real IP](https://nginx.org/en/docs/http/ngx_http_realip_module.html) – Determine true origin IP address for proxied traffic - [SSI](https://nginx.org/en/docs/http/ngx_http_ssi_module.html) – Process Server Side Includes (SSI) commands - [User ID](https://nginx.org/en/docs/http/ngx_http_userid_module.html) – Set cookies that uniquely identify clients - [WebDAV](https://nginx.org/en/docs/http/ngx_http_dav_module.html) – Implement WebDAV file management ### HTTP Access Control and Authentication - [Access](https://nginx.org/en/docs/http/ngx_http_access_module.html) – Control access based on client IP address (support access control lists [ACLs]) - [Auth Basic](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) – Implement HTTP Basic Authentication scheme - [Auth JWT](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html) – Validate JSON Web Tokens - [Auth Request](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html) – Determine client authorization using subrequests to external authentication server - [Auth Require](https://nginx.org/en/docs/http/ngx_http_auth_require_module.html) – Variable-based access control support - [OIDC](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) – Implement authentication as a Relying Party in OpenID Connect solution - [Referer](https://nginx.org/en/docs/http/ngx_http_referer_module.html) – Control access based on `Referer` field in HTTP request header - [Secure Link](https://nginx.org/en/docs/http/ngx_http_secure_link_module.html) – Process encrypted, time-limited links to content ### HTTP Advanced Configuration - [Browser](https://nginx.org/en/docs/http/ngx_http_browser_module.html) – Create variables based on `User-Agent` field in HTTP request header - [Cache Slice](https://nginx.org/en/docs/http/ngx_http_slice_module.html) – Create byte-range segments of large files, for more efficient caching - [Geo](https://nginx.org/en/docs/http/ngx_http_geo_module.html) – Create variables based on client IP address - [Map](https://nginx.org/en/docs/http/ngx_http_map_module.html) – Create variables based on other variables in requests - [Num_map](https://nginx.org/en/docs/http/ngx_http_num_map_module.html) – Create variables whose values depend on numeric values or numeric value ranges - [Rewrite](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html) – Test and change URI of request - [Split Clients](https://nginx.org/en/docs/http/ngx_http_split_clients_module.html) – Partition clients for A/B testing - [Sub](https://nginx.org/en/docs/http/ngx_http_sub_module.html) – Replace text string in response (rewrite content) ### HTTP Logging - [Log](https://nginx.org/en/docs/http/ngx_http_log_module.html) – Log HTTP transactions locally or to `syslog` - [Session Log](https://nginx.org/en/docs/http/ngx_http_session_log_module.html) – Log HTTP transactions aggregated per session ### HTTP Media Delivery - [F4F](https://nginx.org/en/docs/http/ngx_http_f4f_module.html) – Stream HDS (Adobe HTTP Dynamic Streaming; filename extensions **.f4f**, **.f4m**, **.f4x**) - [FLV](https://nginx.org/en/docs/http/ngx_http_flv_module.html) – Stream FLV (Flash Video; filename extension **.flv**) - [HLS](https://nginx.org/en/docs/http/ngx_http_hls_module.html) – Stream HLS (Apple HTTP Live Streaming; filename extensions **.m3u8**, **.ts**) dynamically generated from MP4 or MOV (filename extensions **.m4a**, **.m4v**, **.mov**, **.mp4**, and **.qt**) - [MP4](https://nginx.org/en/docs/http/ngx_http_mp4_module.html) – Stream MP4 (filename extensions **.m4a**, **.m4v**, **.mp4**) - Streaming of RTMP and DASH is provided by the third-party [RTMP](https://github.com/arut/nginx-rtmp-module) module ### HTTP Proxying - [FastCGI](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html) – Proxy and cache requests to FastCGI server - [gRPC](https://nginx.org/en/docs/http/ngx_http_grpc_module.html) – Proxy requests to gRPC server - [Memcached](https://nginx.org/en/docs/http/ngx_http_memcached_module.html) – Proxy requests to memcached server - [Mirror](https://nginx.org/en/docs/http/ngx_http_mirror_module.html) – Send copy of requests to one or more additional servers - [Proxy](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) – Proxy and cache requests to HTTP server - [SCGI](https://nginx.org/en/docs/http/ngx_http_scgi_module.html) – Proxy and cache requests to SCGI server - [Tunnel](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html) – Handle CONNECT requests and establishes an end-to-end virtual connection for forward proxying - [Upstream](https://nginx.org/en/docs/http/ngx_http_upstream_module.html) – Proxy and cache requests to load-balanced pool of servers - [Upstream Health Checks](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html) – Verify servers in load-balanced pool are operational - [uwsgi](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html) – Proxy and cache requests to uwsgi server ### HTTP Transaction Shaping - [Limit Connections](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html) – Limit concurrent connections from a client IP address or other keyed value - [Limit Requests](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html) – Limit rate of request processing for a client IP address or other keyed value - [Limit Responses](https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate) – Limit rate of responses per client connection ### HTTP/2, HTTP/3 and SSL/TLS - [HTTP/2](https://nginx.org/en/docs/http/ngx_http_v2_module.html) – Process HTTP/2 traffic - [HTTP/3](https://nginx.org/en/docs/http/ngx_http_v3_module.html) – Process HTTP/3 traffic - [SSL/TLS](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) – Process HTTPS traffic ### Mail - [Mail Core](https://nginx.org/en/docs/mail/ngx_mail_core_module.html) – Proxy mail traffic - [Auth HTTP](https://nginx.org/en/docs/mail/ngx_mail_auth_http_module.html) – Offload authentication processing from HTTP server - [IMAP](https://nginx.org/en/docs/mail/ngx_mail_imap_module.html) – Implement capabilities and authentication methods for IMAP - [POP3](https://nginx.org/en/docs/mail/ngx_mail_pop3_module.html) – Implement authentication methods for POP3 traffic - [Proxy](https://nginx.org/en/docs/mail/ngx_mail_proxy_module.html) – Support proxy-related parameters for mail protocols - [SMTP](https://nginx.org/en/docs/mail/ngx_mail_smtp_module.html) – Define accepted SASL authentication methods for SMTP clients - [SSL/TLS](https://nginx.org/en/docs/mail/ngx_mail_ssl_module.html) – Implement SSL, STARTTLS, and TLS for mail protocols ### Programmability and Monitoring - [NGINX Plus API](https://nginx.org/en/docs/http/ngx_http_api_module.html) – Provide REST API for accessing metrics, configuring upstream server groups dynamically, and managing key-value pairs, without the need to reload NGINX configuration - [Key-Value Store](https://nginx.org/en/docs/http/ngx_http_keyval_module.html) – Create variables with values taken from key-value pairs managed by the [NGINX Plus API](https://nginx.org/en/docs/http/ngx_http_api_module.html#http_keyvals_) - [Management](https://nginx.org/en/docs/ngx_mgmt_module.html) – [Configure licensing and usage reporting](solutions/about-subscription-licenses.md) of NGINX Plus installation to F5 licensing endpoint or [NGINX Instance Manager](nim/index.md) ### TCP and UDP Proxying and Load Balancing - [Stream](https://nginx.org/en/docs/stream/ngx_stream_module.html) – Process TCP and UDP traffic - [Access](https://nginx.org/en/docs/stream/ngx_stream_access_module.html) – Support IP-based access control lists (ACLs) - [Geo](https://nginx.org/en/docs/stream/ngx_stream_geo_module.html) – Create variables based on client IP address - [Limit Conn](https://nginx.org/en/docs/stream/ngx_stream_limit_conn_module.html) – Limit concurrent connections by key - [Log](https://nginx.org/en/docs/stream/ngx_stream_log_module.html) – Log TCP and UDP transactions - [Map](https://nginx.org/en/docs/stream/ngx_stream_map_module.html) – Create variables based on other variables in requests - [Num_map](https://nginx.org/en/docs/stream/ngx_stream_num_map_module.html) – Create variables whose values depend on numeric values or numeric value ranges - [MQTT Preread](https://nginx.org/en/docs/stream/ngx_stream_mqtt_preread_module.html) – Forward MQTT traffic without processing it - [MQTT Filter](https://nginx.org/en/docs/stream/ngx_stream_mqtt_filter_module.html) – Process Message Queuing Telemetry Transport protocol (MQTT) protocol - [Proxy](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html) – Proxy requests to TCP and UDP servers - [Pass](https://nginx.org/en/docs/stream/ngx_stream_pass_module.html) – Pass any accepted client connection to any configured listening socket in http, stream, mail, and other similar modules - [Real IP](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html) – Determine true origin IP address for proxied traffic - [Return](https://nginx.org/en/docs/stream/ngx_stream_return_module.html) – Return specified value to client and close connection - [Split Clients](https://nginx.org/en/docs/stream/ngx_stream_split_clients_module.html) – Partition clients for A/B testing - [SSL/TLS](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html) – Process TCP traffic secured with SSL/TLS - [SSL/TLS Preread](https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html) – Forward TCP traffic secured with SSL/TLS without decrypting it - [Upstream](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html) – Proxy and cache traffic to load-balanced pool of servers - [Upstream Health Checks](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html) – Verify servers in load-balanced pool are operational -------------------------------------------------------------------------------- title: "Using NGINX or NGINX Plus as the Ingress Controller for Amazon Elastic Kubernetes Services" url: https://docs.nginx.com/nginx/deployment-guides/amazon-web-services/ingress-controller-elastic-kubernetes-services/index.md description: Use NGINX or F5 NGINX Plus as the Ingress Controller for Amazon Elastic Kubernetes Services. -------------------------------------------------------------------------------- This guide explains how to use NGINX Open Source or F5 NGINX Plus with NGINX Ingress Controller for Amazon Elastic Kubernetes Services (EKS). **Note:** These instructions apply to NGINX Ingress Controller with NGINX Open Source or NGINX Plus. For ease of reading, the document refers to NGINX Plus only. ## Prerequisites - [An AWS account](https://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/AboutAWSAccounts.html). - A prebuilt image of the NGINX or NGINX Plus Ingress Controller for Kubernetes. - For NGINX Open Source you can use the pre-built image [on DockerHub](https://hub.docker.com/r/nginx/nginx-ingress/). You can also build your own image. - For NGINX Plus, you must [build an image](https://docs.nginx.com/nginx-ingress-controller/installation/build-nginx-ingress-controller/). **Note:** Never upload your NGINX Plus Ingress Controller images to a public repository such as Docker Hub. Doing so violates your license agreement. Run the `make container` command below. ```shell make container DOCKERFILE=DockerfileForPlus PREFIX=nginx/nginx-plus-ingress ``` The `PREFIX` argument specifies the repo name in your private container registry. In this example, we set it to `nginx/nginx-plus-ingress`. You can later use that name to reference the image instead of its numerical ID. ## Create an Amazon EKS Cluster You can create an Amazon EKS cluster with: - the AWS Management Console - the AWS CLI - the `eksctl` command line utility. This guide covers the `eksctl` command as it is the simplest option. 1. Follow the instructions in the [eksctl.io documentation](https://eksctl.io/installation/) to install or update the `eksctl` command. 2. Create an Amazon EKS cluster by following the instructions in the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html). Select the **Managed nodes – Linux** option for each step. Note that the `eksctl create cluster` command in the first step can take ten minutes or more. ## Push the NGINX Plus Ingress Controller Image to AWS ECR This step is only required if you do not plan to use the prebuilt NGINX Open Source image. 1. Use the [AWS documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html) to create a repository in the Amazon Elastic Container Registry (ECR). In Step 4 of the AWS instructions, name the repository **nginx-plus-ic** as that is what we use in this guide. 2. Run the following AWS CLI command. It generates an auth token for your AWS ECR registry, then pipes it into the `docker login` command. This lets AWS ECR authenticate and authorize the upcoming Docker requests. For details about the command, see the [AWS documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html). ```shell aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com ``` - `` is the same region name you specified in Step 2 above. - `` is your AWS account number. For instructions on retrieving the ID, see the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html). 3. Run the following command to apply the tag `edge` to your NGINX Ingress Controller image: ```shell docker tag :edge .dkr.ecr..amazonaws.com/:edge ``` - `` is the repo name you set with the `PREFIX` parameter to the `make container` command (see [Prerequisites](#prereqs)). In this guide it is `nginx/nginx-plus-ingress`. - `` is the AWS ECR repository you created in Step 1 above. In this guide it is called `nginx-plus-ic`. The final command is: ```shell docker tag nginx/nginx-plus-ingress:edge .dkr.ecr..amazonaws.com/nginx-plus-ic:edge ``` 5. Push the NGINX Plus Ingress Controller image to AWS ECR: ```shell docker push .dkr.ecr..amazonaws.com/:edge ``` ## Install the NGINX Plus Ingress Controller Use [our documentation](https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/) to install the NGINX Plus Ingress Controller in your Amazon EKS cluster. Complete the steps up to and including [Confirm NGINX Ingress Controller is running](https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-manifests/#confirm-nginx-ingress-controller-is-running). Next, follow the instructions below to create a Network Load Balancer to route traffic to NGINX Plus Ingress Controller. --- ## Use a Network Load Balancer in front of NGINX Ingress Controller These steps assume you've cloned the [kubernetes-ingress](https://github.com/nginx/kubernetes-ingress) repository in the previous step. You need a Kubernetes `LoadBalancer` service to route traffic to the NGINX Ingress Controller. By default, Amazon EKS will create a [Classic Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/introduction.html) for Kubernetes services of type `LoadBalancer`. However, we recommend that you create a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) (NLB). It operates at the transport layer and is optimized for high performance and low latency. We also recommend enabling the PROXY Protocol for both the NGINX Plus Ingress Controller and your NLB target groups. This is used to forward client connection information. If you choose not to enable the PROXY protocol, see the [Appendix](#appendix). ### Configure a `LoadBalancer` Service to Use NLB Apply the manifest `deployments/service/loadbalancer-aws-elb.yaml` to create a `LoadBalancer` of type NLB: ```shell kubectl apply -f deployments/service/loadbalancer-aws-elb.yaml ``` ### Enable the PROXY Protocol 1. Add the following keys to the `deployments/common/nginx-config.yaml` config map file: ```yaml proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" ``` 2. Run the following command to update the config map. This will enable the PROXY protocol for the NGINX Plus Ingress Controller: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` 3. Enable the PROXY Protocol for the target groups linked to the NLB. Follow the steps in the **Enable proxy protocol** section of the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html#proxy-protocol) to do this. ## Testing 1. Get the DNS name of NGINX Ingress Controller with the command below. Find the value of the "LoadBalancer Ingress" in the output. ```shell kubectl describe svc nginx-ingress --namespace=nginx-ingress ``` 2. Resolve the DNS name into an IP address using `nslookup`: ```shell nslookup ``` 3. Follow the [instructions](https://github.com/nginx/kubernetes-ingress/tree/main/examples/ingress-resources/complete-example) to deploy the Cafe demo app into the EKS cluster. It will be load balanced by NGINX Ingress Controller. * In Step 1 of deploying the demo app, save the public IP address into the `IC_IP` shell variable. Set `IC_HTTPS_PORT` to 443. * The `kubectl` commands are relative to the `deployment/examples/ingress-resources/complete-example` directory of the [kubernetes-ingress](https://github.com/nginx/kubernetes-ingress) repository. * Run the `curl` command listed in the instructions. It will access the demo app and populate the NGINX Plus Ingress Controller logs. 4. Run the following commands to check if the PROXY Protocol is enabled: 1. Display the pod of NGINX Ingress Controller: ```shell kubectl get pods -n nginx-ingress ``` 2. Display the logs from NGINX Ingress Controller. Replace `` with the name from the previous step. If the logged IP address matches the one you used to access the demo app, then the PROXY Protocol is enabled. ```shell kubectl logs -n nginx-ingress ``` ## Appendix: Disable the PROXY Protocol If you want to disable the PROXY Protocol, perform these steps. 1. Disable the PROXY Protocol for the target groups linked to the NLB. Undo the steps in the **Enable proxy protocol** section of the [AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html#proxy-protocol). 2. Remove the following keys from `deployments/common/nginx-config.yaml` the config map file: ```yaml proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" ``` 2. Run the following command to update the config map: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` 3. In the `deployments/service/loadbalancer-aws-elb.yaml` service file, add the `externalTrafficPolicy` key in the `spec` section. Set it to `Local`, as in this example: ```yaml apiVersion: v1 kind: Service metadata: name: nginx-ingress-nlb namespace: nginx-ingress annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp" service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" service.beta.kubernetes.io/aws-load-balancer-type: nlb spec: externalTrafficPolicy: Local type: LoadBalancer ports: - port: 80 targetPort: 80 protocol: TCP name: http - port: 443 targetPort: 443 protocol: TCP name: https selector: app: nginx-ingress ``` 4. Run the following command to update the service: ```shell kubectl apply -f deployments/service/loadbalancer-aws-elb.yaml ``` ### Revision History - Version 2 (February 2025) – Initial version (NGINX Plus Release 20) -------------------------------------------------------------------------------- title: "How NGINX Plus Performs Zone Synchronization" url: https://docs.nginx.com/nginx/admin-guide/high-availability/zone_sync_details/index.md description: Design and limitations of the zone synchronization feature that allows synchronizing data in an nginx cluster. -------------------------------------------------------------------------------- This chapter describes the design and limitations of the zone synchronization feature that allows synchronizing data in an nginx cluster. ## Introduction ### Use Cases Given the specific functionality of the feature, namely: - processing requests on each node separately without consulting other nodes - eventual delivery of changes - expirable records The following use cases are considered: - **Session caching**: the first request creates a session which is shared with other nodes. Subsequent requests use the same session on any cluster node. A missing session on a node is more of a performance penalty rather than a fatal error. - **Resource limiting**: each node assesses local resources and notifies others. In average, the cluster limit is properly imposed. - **Dynamic configuration**: dynamic configuration entries (for example, temporary redirection rules, access rules and limits) are automatically shared within the cluster after being entered on one node. As soon as other nodes receive them, new rules are applied. ## Modules The feature is complex, and its support is split between the nginx `core`, the [ngx_stream_zone_sync_module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) module, and the modules that actually use shared zones (for example, [sticky](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky) or [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html) modules; here, we will refer to such modules as `functional`). Here, the [zone synchronization module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) is responsible for: - all network-related functionality, including: - cluster nodes discovery - managing network connections to cluster nodes - providing transport layer security (using the [ssl module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html)) - pushing data provided by the functional modules to remote nodes - accepting data from remote nodes and dispatching it to the functional modules - tracking changes in zones and processing the zone’s output queue On the server side, the [zone synchronization module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) is a regular stream module, so access control, transport layer security, and limiting are supported. The functional modules are responsible for: - serializing zone records so they can be sent over the network along with key and timestamp metadata - deserializing messages and updating the local zone with remote data - maintaining a queue of records that were updated locally and have to be distributed within the cluster The core provides necessary interfaces between the zone synchronization module and the functional modules. ## Synchronization ### Data Model Each functional module represents its data as a queue of records. Records are opaque for the zone synchronization module; it deals only with a serialized representation of the records while sending them to other nodes. The queue contains records that need to be sent to the cluster. Only records created locally on a node may appear in the output queue. The queue is intrusive and may include all records stored in a zone. The zone synchronization module polls the output queues of configured zones at a constant [interval](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_interval) which controls how large the inconsistency window of the cluster is. Also, it allows reducing the amount of data transferred in case of frequent updates of a single record (a typical use case with sessions, when a session is created and later updated multiple times). Synchronized records may be `created` or `updated`. Currently, the `delete` operation is not supported; instead, each record has a configuration-defined lifetime. When a record expires, it is deleted locally, and no updates are sent to the cluster. The functional modules control whether the record lifetime is extended upon access/update or not. The data becomes consistent `eventually`: there is always a time span when an updated value varies between the cluster nodes. The span can be reduced (but not eliminated) by tuning the [zone synchronization interval](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_interval). In terms of the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem), the zone sync cluster can be categorized as an AP system. Consistency is ensured by the fact that each node eventually receives all events that have happened in the cluster ordered by timestamp, processes them using a common algorithm, and arrives at the same state as other nodes. Obviously, the algorithm depends on proper temporal synchronization of all cluster nodes, so it is important to setup NTP or similar technology on all nodes. ### Data flow 1. The functional module creates a new record (for example, a new sticky session) 2. The record is inserted into the output queue 3. The functional module may continue its normal operation. As a result, the record can be updated multiple times or even deleted (for example, locally on the node after a direct request) before being shared with other nodes. 4. Periodic timer triggers, and the zone synchronization module inspects the output queue of the functional module. 5. If the queue is not empty, the zone synchronization module asks the functional module to serialize the outgoing records and writes them into its own [buffer](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_buffers). 6. The functional module serializes a record using its own serialization format. Typically, it includes `key`, `timestamp`, and `payload`. 7. The zone synchronization module consumes the output queue until it has no buffer space available or the queue is empty. 8. Finally, the zone synchronization module sends all information it has read to nodes it has established connections with, framing the records into messages that contain zone name and version details. If an error occurs and a record cannot be delivered to a node, the connection to the node is closed to be reestablished. See the explanation of the initial state of a cluster portion below. 9. The remote node receives the message and dispatches it to the appropriate functional module and zone. The functional module now has to refresh its local state using the information from the remote node (`key`, `timestamp`, and `payload`) by: - Inserting the new record at the key if no such key exists - Updating an existing record if the remote timestamp is newer - Ignoring the update if the local timestamp is newer 10. The functional module, that is capable to serialize zone state to disk (i.e. keyval with the “state” enabled) saves record timestamps to ensure proper handling of records lifetime after server restart. ### Topology Topologically, the cluster is a full mesh where each node connects to all the others and any changes are pushed to other nodes. All cluster nodes are equal. There are no special `central` or `main` nodes; each node processes requests without waiting for other nodes. Nodes commit any changes to local memory and continue processing requests. Other nodes will be eventually notified about local changes. A node does not know whether its local values are "best/latest"; it relies on them until updates from other nodes are applied. A node listens for changes from other nodes and applies them as they arrive. ### Initial State When a node is connected to the cluster, it already has a state and it is not known whether any parts of this state were sent to the cluster previously. The ultimate solution is to merge the entire node state with the cluster. This process is called `snapshotting`: when the network connection is established with the new node, the cluster and the node exchange snapshots. The most recent versions of records are selected to form the new joint state. This process runs in parallel with regular operations, so the node does not need to wait until snapshotting ends. The same process happens after a cluster brain split: new nodes appear in the cluster and exchange their current state with peers. If the state is large and there are many nodes in the cluster, synchronization can take some time and introduce additional sporadic system load. When a node goes offline, it temporarily affects other nodes: next data read from the zones is delayed until the write timeout expires, or a connection write error is detected, and the faulty node is disconnected. Then, synchronization continues among the remaining nodes until the connection to the offline node is reestablished. ### Wire Format The module uses a set of [buffers](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_buffers) pre-allocated per node connection to deliver changes. A buffer contains a message header and serialized records. The message header contains: - full message length, including its payload - protocol version (currently `1`) - zone name length - module tag - unique module id - module version - destination zone name The size of the fixed-length header section is `12` bytes. For a message to be successfully dispatched, the signature (module tag and version) must match the destination. Messages with unknown signatures are ignored (for example, adding a new zone to another node can yield such a message). Length of the serialized record depends on the specific functional module; for example, the sticky module generates a `21`-byte fixed-length header and a session ID of up to `32` bytes. If a single serialized record is too large to fit in the buffer, an error is logged and the record is ignored during synchronization. This also suggests that the buffer size should be increased. ### Monitoring The standard functionality of access log allows watching stream sessions with other cluster nodes and log supported variables such as [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr), [`$status`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_status), [`$upstream_bytes_sent`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#var_upstream_bytes_sent) or [`$bytes_received`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#var_bytes_received). The error log registers `NOTICE`-level entries with details of various events: node discovery, node connection, accepted client, and others. Occurring errors are reported at the appropriate level. The zone synchronization module exports several counters via the [API](https://nginx.org/en/docs/http/ngx_http_api_module.html) module: the [`stream/zone_sync`](https://nginx.org/en/docs/http/ngx_http_api_module.html#stream_zone_sync_) endpoint includes the [status](https://nginx.org/en/docs/http/ngx_http_api_module.html#def_nginx_stream_zone_sync) endpoint with per-instance information; [`zones/`](https://nginx.org/en/docs/http/ngx_http_api_module.html#def_nginx_stream_zone_sync_zone) endpoints list statistics per zone. The [`nodes online`](https://nginx.org/en/docs/http/ngx_http_api_module.html#def_nginx_stream_zone_sync_zone) metric displays the count of established connections to other nodes. Usually, it is expected to equal `N` minus `1` where `N` is the number of all cluster nodes. Additionally, the endpoint contains cumulative counters of inbound and outbound messages and bytes. The per-zone statistics include the total number of records in the shared memory zone and the number of records in the output queue. ### Scaling Note that with synchronization your nodes start receiving multiple updates from other nodes which typically requires locking a certain amount of shared memory to insert/update records. With many workers and nodes, this can become a bottleneck. If the number of events is high, consumed network bandwidth and CPU usage increase accordingly. ## FAQ **Q.** What happens when a node is isolated from the cluster? **A.** When a node is isolated from the cluster, it continues to operate as a single node: i.e. it manages sessions locally and continues to respond to clients, while continually trying to connect to others. **Q.** What happens when an isolated node reconnects to the cluster? **A.** When a node reconnects, it performs full resync: it sends all local sessions to cluster, and, in turn, receives data from other nodes.Thus, when all nodes are connected, they will eventually reach a consistent state. **Q.** If multiple nodes sync with the same data, how are conflicts resolved? **A.** The conflict resolution is based on time, thus it is critical to have synchronized clocks across the cluster. If two nodes created a record with same key, the newest record wins. **Q.** Is there a state file for cluster sync data? **A.** No. Cluster sync data is shared across the cluster so there is no need to write it out to disk. The keyval module is capable of saving local node state to disk. **Q.** Can F5 NGINX Plus be configured to reject connections before sync has completed? **A.** No, because there is no such state. Each node just receives stream of updates from others and sends own updates to others. If no new data is arriving to cluster, all nodes will have an empty backlog queue and this can be counted as 'complete sync', but in reality such situation is usually never met: there are always clients. **Q.** How do I deliberately put a node offline? **A.** The documentation describes how to control cluster nodes. **Q.** How can we monitor cluster state data? **A.** There are 2 counters available for each synchronized zone in the NGINX Plus API: Total number of records on node, and a number of records that needs to be sent. If all nodes have approximately the same number of records and almost empty outgoing queue, we may consider the cluster to be healthy. **Q.** How can we monitor cluster health? **A.** There are 2 further metrics exposed by the NGINX Plus API: 1. Number of connected nodes: this is expected to be equal on all nodes and be equal to total number of nodes in cluster minus one. That is, you can monitor cluster connectivity: if some nodes are dead/disconnected, this counters will change 2. Length of send queue: ideally, this is zero (low load) or some constant number (average load). If a backlog is growing and you did not made changes (i.e. adding or removing nodes to cluster), this indicates problems (network connection issues, dead nodes, etc) 3. As usual, monitor error logs: all related events are logged. By default, only errors are logged. It is possible to view more cluster events at INFO level. ## History In NGINX Plus [R15](nginx/releases.md#r15), the ability to synchronize memory zones between instances was introduced; only sticky sessions with [learn](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_learn_sync) method were supported at the moment. In NGINX Plus [R16](nginx/releases.md#r16), shared zones synchronization was extended to support [keyval](https://nginx.org/en/docs/http/ngx_http_keyval_module.html) and [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) modules. In NGINX Plus [R18](nginx/releases.md#r18), a single zone_sync configuration can be applied between all instances in a cluster using wildcards in the [listen](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#listen) directive. -------------------------------------------------------------------------------- title: "Installing NGINX Plus on Microsoft Azure" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus-microsoft-azure/index.md description: Install F5 NGINX Plus as a virtual machine image on Microsoft Azure, to provide sophisticated Layer 7 load balancing for your apps. -------------------------------------------------------------------------------- [F5 NGINX Plus](https://www.f5.com/products/nginx/nginx-plus), the high-performance application delivery platform, load balancer, and web server, is available at the Microsoft Azure Marketplace as a virtual machine (VM) image. The VM image contains the latest version of NGINX Plus, optimized for use with Azure. ## Install the NGINX Plus VM To quickly set up an NGINX Plus environment on Microsoft Azure: 1. Follow the instructions in [Create a Virtual Machine Running Linux](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal) to sign up on Azure and get more information about Azure itself. 2. Search for “NGINX Plus” in the [Azure Marketplace](https://marketplace.microsoft.com/en-us/search/products?search=nginx%2C%20inc&page=1), open the VM image, and follow the installation instructions. 3. Create an Azure _availability set_ of two or more NGINX Plus virtual machines, which adds redundancy to your NGINX Plus setup by ensuring that at least one virtual machine remains available during a planned or unplanned maintenance event on the Azure platform. For more information, see [Manage the availability of Linux virtual machines](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/manage-availability?) in the Azure documentation. 4. Create _endpoints_ to enable clients outside the NGINX Plus virtual machine’s cloud or virtual network to access it. Sign in to the Azure Management Portal and add endpoints manually to handle the inbound network traffic on port 80 (HTTP) and port 443 (HTTPS). For more information, see [How to set up endpoints on a Linux classic virtual machine in Azure](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/classic/setup-endpoints) in the Azure documentation. 5. As soon as the new virtual machine launches, NGINX Plus starts automatically and serves a default **index.html** page. To verify that NGINX Plus is working properly, use a web browser to access the public DNS name of the new virtual machine and view the page. You can also check the status of the NGINX Plus server by logging into the virtual machine and running this command: ```none /etc/init.d/nginx status ``` ## What If I Need Help? If you encounter any problems with NGINX Plus configuration, documentation is available at [nginx.org](https://nginx.org/en/docs/) and in the [NGINX Plus Admin Guide](/nginx/admin-guide/installing-nginx/). Customers who purchase an NGINX Plus VM image at the Azure Marketplace are eligible for the Azure VM support provided by the NGINX, Inc. engineering team. To activate support, submit the [Azure support activation](https://www.nginx.com/azure-support-activation/) form (you need your Azure subscription ID). When you request support, we’ll ask you to provide the Azure subscription ID that you registered, along with the deployment IDs of your Azure virtual machines in some cases. -------------------------------------------------------------------------------- title: "Monitoring NGINX and NGINX Plus with the New Relic Plug-In" url: https://docs.nginx.com/nginx/admin-guide/monitoring/new-relic-plugin/index.md description: Find what's new in version 2 of the NGINX and F5 NGINX Plus plug‑in for New Relic – more convenience, more metrics, richer visualization. -------------------------------------------------------------------------------- In March, 2013 we released the first version of the [“nginx web server” plug‑in](https://newrelic.com/instant-observability/nginx) for New Relic monitoring of the NGINX Open Source software and F5 NGINX Plus. Since then, we’ve received lots of interest from users – we greatly appreciate it! The plug‑in continues to be one of the most popular New Relic plug‑ins, in spite of the few things that (we believe!) could be improved. If you don’t already have a New Relic account, [sign up](http://newrelic.com/). We selected Ruby as the base language for the original plug‑in, because the New Relic platform itself is written in Ruby, the API was not yet published, and many things were changing quickly. Later, after the launch of the New Relic platform, we continued to work on adding a number of improvements based on users’ feedback and our own vision of how to improve the overall usability of the NGINX monitoring plug‑in, and realized the need for further changes. ## What’s New? Today, we are pleased to announce a major update of the NGINX plug‑in for New Relic, **version 2.0**, which includes the following changes: - The plug‑in is rewritten in Python. You no longer need to install Ruby. - The plug‑in is finally packaged. There are prebuilt packages for RHEL/CentOS‑based and Debian/Ubuntu‑based systems. An included init script enables you to easily set up autostart of the plug‑in. - There are two new sections in the live activity monitoring dashboard, for NGINX Plus customers: - **Servers** – Additional summary counters for virtual servers whose `server` configuration block includes the [status_zone](https://nginx.org/en/docs/http/ngx_http_api_module.html#status_zone) directive - **Cache** – Cumulative stats for all configured caches - Verbose logging is enabled by default. ## Installation Download the [plug‑in and installation instructions](https://docs.newrelic.com/install/nginx/). ## Configuring the Plug‑In The configuration file for the NGINX plug‑in is **/etc/nginx-nr-agent/nginx-nr-agent.ini**. The minimal configuration includes: - Your New Relic license key in the `newrelic_license_key` statement in the `global` section. - At least one `source` section. The name of the section is used in log entries only, and can contain almost any character string you want. Two parameters are required: - `name` – NGINX instance name in the New Relic UI. - `url` – Full URL to the corresponding instance. The plug‑in accepts source data in the format generated when the [stub_status](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status) directive is included in the configuration of NGINX or NGINX Plus. It also supports the JSON‑formatted output generated when the [api](https://nginx.org/en/docs/http/ngx_http_api_module.html#api) directive is included in an NGINX Plus configuration. You can include the optional `http_user` and `http_pass` statements to set HTTP basic authentication credentials in cases where the corresponding location is protected by the NGINX [auth_basic](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html#auth_basic) directive. The default log file is **/var/log/nginx-nr-agent.log**. ## Running the Plug‑In You can manage the plug‑in with an init script. Run the `service` `nginx-nr-agent` command with the `start`, `stop`, or `status` argument to start, stop, or display the status of the plug‑in, respectively. When started from an init script, the plug‑in daemon runs as the `nobody` user. You can start the plug‑in directly by running `/usr/bin/nginx-nr-agent.py`, with the following optional parameters: - `-c`, `--config` – Path to configuration file - `-p`, `--pidfile`> – Path to PID file - `-f`, `--foreground` – Do not detach from terminal (useful for debugging) If everything is working as expected, the plug‑in appears as a tab in the New Relic UI: [![image of NGINX instance in New Relic instances list](/nginx/images/0s-instances.png)](/nginx/images/0s-instances.png) ## Dashboard Examples ### Overview The **Overview** tab displays the most basic metrics for the whole instance: number of active and idle connections, and request rate. [![image of Overview tab in New Relic UI](/nginx/images/1s-overview.png)](/nginx/images/1s-overview.png) ### Connections The **Connections** tab displays various metrics about client connections. [![image of Connections tab in New Relic UI](/nginx/images/2s-connections.png)](/nginx/images/2s-connections.png) ### Requests The **Requests** tab displays the number of requests currently being processed and the overall request rate. [![image of Requests tab in New Relic UI](/nginx/images/3s-requests.png)](/nginx/images/3s-requests.png) ### Upstreams (NGINX Plus only) The **Upstreams** tab displays detailed statistics about all configured upstream groups. [![image of Upstreams tab in New Relic UI](/nginx/images/4s-upstreams.png)](/nginx/images/4s-upstreams.png) ### Servers (NGINX Plus only) The **Servers** tab displays detailed statistics about all configured virtual server zones. [![image of Servers tab in New Relic UI](/nginx/images/5s-servers.png)](/nginx/images/5s-servers.png) ### Cache (NGINX Plus only) The **Cache** tab displays summary statistics for all configured caches. [![image of Caches tab in New Relic UI](/nginx/images/6s-cache.png)](/nginx/images/6s-cache.png) ## What’s Next? We plan to extend the plug‑in’s functionality with additional metrics for both NGINX and NGINX Plus, and your feedback is very important to us. Ideas, thoughts, questions? Let us know by commenting on this post. Thanks for using NGINX and NGINX Plus! To try NGINX Plus, start your [free 30-day trial](https://www.nginx.com/free-trial-request/) today or [contact us](https://www.nginx.com/contact-sales/) for a demo. -------------------------------------------------------------------------------- title: "Open Source Components" url: https://docs.nginx.com/nginx/open-source-components/index.md description: License information for open source components included in the NGINX Plus software. -------------------------------------------------------------------------------- Open source components included in the F5 NGINX Plus (package name is `nginx-plus`) are: - nginx/OSS 1.29.3, distributed under 2-clause BSD license. Homepage: Copyright © 2002-2021 Igor Sysoev Copyright © 2011-2025 NGINX, Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - Internal MD5 implementation based on Alexander Peslyak's public domain implementation: This is an OpenSSL-compatible implementation of the RSA Data Security, Inc. MD5 Message-Digest Algorithm (RFC 1321). Homepage: Author: Alexander Peslyak, better known as Solar Designer This software was written by Alexander Peslyak in 2001. No copyright is claimed, and the software is hereby placed in the public domain. In case this attempt to disclaim copyright and place the software in the public domain is deemed null and void, then the software is Copyright © 2001 Alexander Peslyak and it is hereby released to the general public under the following terms: - Redistribution and use in source and binary forms, with or without modification, are permitted. - There's ABSOLUTELY NO WARRANTY, express or implied. (This is a heavily cut-down "BSD license".) - MurmurHash algorithm (version 2), distributed under MIT license. Homepage: Copyright © Austin Appleby Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - Components used in status monitoring dashboard v2 only (`dashboard.html` in `nginx-plus` package) and distributed under MIT license: - `@babel-core`, Babel compiler core (7.23.2) Homepage: Copyright © 2014-present Sebastian McKenzie and other contributors - `@babel/plugin-proposal-object-rest-spread`, produces spec-compliant code by using Babel's objectSpread helper (7.22.15). Homepage: - `@babel/plugin-transform-runtime`, makes helpers reference the module babel-runtime to avoid duplication across your compiled output (7.22.15). Homepage: - `@babel/preset-env`, a Babel preset for each environment (7.22.15). Homepage: - `@babel/preset-react`, a Babel preset for all React plugins (7.22.15). Homepage: - `autoprefixer`, a PostCSS plugin to parse CSS and add vendor prefixes to CSS rules (10.4.7) Homepage: Copyright © 2013 Andrey Sitnik - `babel-loader`, allows transpiling JavaScript files using Babel and webpack (9.1.3). Homepage: Copyright © 2014-2019 Luís Couto - `babel-plugin-istanbul`, a babel plugin that adds istanbul instrumentation to ES6 code (6.1.1). Homepage: Copyright © 2016, Istanbul Code Coverage - `core-js`, a modular standard library for JavaScript (3.22.8). Homepage: Copyright © 2014-2022 Denis Pushkarev - `css-loader`, interprets `@import` and `url()` like `import/require()` and will resolve them (6.8.1). Homepage: Copyright © JS Foundation and other contributors - `cssnano`, a modular minifier, built on top of the PostCSS ecosystem (5.1.11). Homepage: Copyright © Ben Briggs - `eslint`, a tool for identifying and reporting on patterns found in ECMAScript/JavaScript code (8.49.0). Homepage: Copyright OpenJS Foundation and other contributors, - `eslint-config-airbnb`, exports some ESLint configurations (19.0.4). Homepage: Copyright © 2012 Airbnb - `eslint-plugin-import`, supports linting of ES2015+ (ES6+) import/export syntax and prevent issues with misspelling of file paths and import names (2.28.1). Homepage: Copyright © 2015 Ben Mosher - `eslint-plugin-jsx-a11y`, static AST checker for accessibility rules on JSX elements (6.5.1). Homepage: Copyright © 2016 Ethan Cohen - `eslint-plugin-react`, react specific linting rules for eslint (7.33.2). Homepage: Copyright © 2014 Yannick Croissant - `eslint-webpack-plugin`, uses eslint to find and fix problems in the JavaScript code (4.0.1). Homepage: Copyright JS Foundation and other contributors - `history`, manage session history with JavaScript (4.10.1). Homepage: Copyright © React Training 2016-2020, Copyright © Remix Software 2020-2021 - `html-inline-css-webpack-plugin`, converts external stylesheet to embedded stylesheet, aka document stylesheet (1.11.1). Homepage: Copyright © 2018 Huang - `html-webpack-plugin`, simplifies creation of HTML files to serve your webpack bundles (5.5.0). Homepage: Copyright © JS Foundation and other contributors - `mini-css-extract-plugin`, extracts CSS into separate files (2.6.0). Homepage: Copyright © JS Foundation and other contributors - `postcss`, a tool for transforming styles with JS plugins (8.4.31). Homepage: Copyright 2013 Andrey Sitnik <> - `postcss-loader`, PostCSS loader for webpack (7.3.3). Homepage: Copyright © JS Foundation and other contributors - `postcss-url`, PostCSS plugin to rebase url(), inline or copy asset (10.1.3). Homepage: Copyright © 2014 Maxime Thirouin - `preact`, fast 3kb React alternative with the same ES6 API (10.7.3). Homepage: Copyright © 2015-present Jason Miller - `react-dev-utils`, utilities used by Create React App (12.0.1). Homepage: Copyright © 2013-present, Facebook, Inc. - `regenerator-runtime`, standalone runtime for Regenerator-compiled generator and async functions (0.14.1). Homepage: Copyright © 2014-present, Facebook, Inc. - `style-loader`, injects CSS into the DOM (3.3.1). Homepage: Copyright © JS Foundation and other contributors - `webpack`, a bundler for javascript and friends (5.88.2). Homepage: Copyright © JS Foundation and other contributors - `webpack-cli`, provides the interface of options webpack uses in its configuration file (4.10.0). Homepage: Copyright © JS Foundation and other contributors - `whatwg-fetch`, a window.fetch JavaScript polyfill (2.0.4). Homepage: Copyright © 2014-2016 GitHub, Inc. The MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - Components used in status monitoring dashboard v2 only (`dashboard.html` in `nginx-plus` package) and distributed under 3-clause BSD license and Apache 2.0 license: - `babel-plugin-react-css-modules`, transforms styleName to className using compile time CSS module resolution (3.4.2), distributed under 3-clause BSD license. Homepage: Copyright © 2016, Gajus Kuizinas () All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - Neither the name of the Gajus Kuizinas () nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ANUARY BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - `npm-font-open-sans`, Open Sans font family - incl. usage of CSS, SCSS, LESS (1.1.0), distributed under Apache 2.0 license. Homepage: Copyright © Steve Matteson Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Optional add-on and third-party modules provided with NGINX Plus may include additional open-source components. The licenses for these components are included in the installation package for each module. -------------------------------------------------------------------------------- title: "Setting up JWT Authentication" url: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-jwt-authentication/index.md description: This article explains how to control authentication of your web resources using JWT authentication. -------------------------------------------------------------------------------- ## Introduction With F5 NGINX Plus it is possible to control access to your resources using JWT authentication. JWT is data format for user information in the OpenID Connect standard, which is the standard identity layer on top of the OAuth 2.0 protocol. Deployers of APIs and microservices are also turning to the JWT standard for its simplicity and flexibility. With JWT authentication, a client provides a JSON Web Token, and the token will be validated against a local key file or a remote service. ## Prerequisites - NGINX Plus [Release 10](nginx/releases.md#r10) (R10) for native [JWT support](https://www.nginx.com/blog/nginx-plus-r10-released/#r10-jwt) - NGINX Plus [Release 14](nginx/releases.md#r14) for access to [nested JWT claims and longer signing keys](https://www.nginx.com/blog/nginx-plus-r14-released/#jwt) - NGINX Plus [Release 17](nginx/releases.md#r17) for [getting JSON Web keys from a remote location](https://www.nginx.com/blog/nginx-plus-r17-released/#r17-openid) - NGINX Plus [Release 24](nginx/releases.md#r24) (R24) for support of encrypted tokens (JWE) - NGINX Plus [Release 25](nginx/releases.md#r25) (R25) for support of Nested JWT, multiple sources of JSON Web keys, condition-based JWT authentication - NGINX Plus [Release 26](nginx/releases.md#r26) (R26) for support of [JWT key caching](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_cache) - An identity provider (IdP) or service that creates JWT. For manual JWT generation, see "Issuing a JWT to API Clients" section of the [Authenticating API Clients with JWT and NGINX Plus](https://www.nginx.com/blog/authenticating-api-clients-jwt-nginx-plus/) blog post. NGINX Plus supports the following types of JWT: - JSON Web Signature (JWS) - JWT content is digitally signed. The following algorithms can be used for signing: - HS256, HS384, HS512 - RS256, RS384, RS512 - ES256, ES384, ES512 - EdDSA (Ed25519 and Ed448 signatures) - JSON Web Encryption (JWE) - the contents of JWT is encrypted. The following content encryption algorithms (the "enc" field of JWE header) are supported: - A128CBC-HS256, A192CBC-HS384, A256CBC-HS512 - A128GCM, A192GCM, A256GCM The following key management algorithms (the "alg" field of JWE header) are supported: - A128KW, A192KW, A256KW - A128GCMKW, A192GCMKW, A256GCMKW - dir - direct use of a shared symmetric key as the content encryption key - RSA-OAEP, RSA-OAEP-256, RSA-OAEP-384, RSA-OAEP-512 - Nested JWT - support for JWS enclosed into JWE ## Configuring NGINX Plus to Authenticate API Let's assume that NGINX Plus serves as a gateway (`proxy_pass http://api_server`) to a number of API servers (the [`upstream {}`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream) block), and requests passed to the API servers should be authenticated: ```nginx upstream api_server { server 10.0.0.1; server 10.0.0.2; } server { listen 80; location /products/ { proxy_pass http://api_server; #... } } ``` To implement JWT for authentication: 1. First, it is necessary to create a JWT that will be issued to a client. You can use your identity provider (IdP) or your own service to create JWTs. For testing purposes, you can create your own JWT, see [Authenticating API Clients with JWT and NGINX Plus](https://www.nginx.com/blog/authenticating-api-clients-jwt-nginx-plus/) blog post for details. 2. Configure NGINX Plus to accept JWT: specify the [`auth_jwt`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt) directive that enables JWT authentication and also defines the authentication area (or "realm", "API" in the example): ```nginx server { listen 80; location /products/ { proxy_pass http://api_server; auth_jwt "API"; #... } } ``` NGINX Plus can also obtain the JWT from a query string parameter. To configure this, include the `token=` parameter to the [`auth_jwt`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt) directive: ```nginx #... auth_jwt "API" token=$arg_apijwt; #... ``` 3. Specify the type of JWT - `signed` (JWS), `encrypted` (JWE) or `nested` (Nested JWT) - with the [`auth_jwt_type`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_type) directive. The default value of the directive is `signed`, so for JWS, the directive can be omitted. ```nginx server { listen 80; location /products/ { proxy_pass http://api_server; auth_jwt "API"; auth_jwt_type encrypted; #... } } ``` 4. Specify the path to the [JSON Web Key file](#jwk_create) that will be used to verify JWT signature or decrypt JWT content, depending on what you are using. This can be done with the [`auth_jwt_key_file`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) and/or [`auth_jwt_key_request`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directives. Specifying both directives at the same time will allow you to specify more than one source for keys. If none of the directives are specified, JWS signature verification will be skipped. In this scenario, the keys will be taken from two files: the `key.jwk` file and the `keys.json` file: ```nginx server { listen 80; location /products/ { proxy_pass http://api_server; auth_jwt "API"; auth_jwt_type encrypted; auth_jwt_key_file conf/key.jwk; auth_jwt_key_file conf/keys.json; } } ``` In this scenario, there are also two sources for the keys, but the private keys will be taken from the local file `private_jwe_keys.jwk`, while the public keys will be taken from the external identity provider service `https://idp.example.com` in a [subrequest](#auth_jwt_key_request): ```nginx server { listen 80; location /products/ { proxy_pass http://api_server; auth_jwt "API"; auth_jwt_type encrypted; auth_jwt_key_file private_jwe_keys.jwk; auth_jwt_key_request /public_jws_keys; } location /public_jws_keys { proxy_pass "https_//idp.example.com/keys"; } } ``` It is recommended to enable JWT key caching to get the optimal performance from the JWT module. For example, you can use the [`auth_jwt_key_cache`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_cache) directive for the above configuration, and enable the JWT key caching for one hour. Note that if the [`auth_jwt_key_request`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) or [`auth_jwt_key_file`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) are configured dynamically with variables, [`auth_jwt_key_cache`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_cache) cannot be used. ```nginx server { listen 80; location /products/ { proxy_pass http://api_server; auth_jwt "API"; auth_jwt_type encrypted; auth_jwt_key_file private_jwe_keys.jwk; auth_jwt_key_request /public_jws_keys; auth_jwt_key_cache 1h; } location /public_jws_keys { proxy_pass "https_//idp.example.com/keys"; } } ``` ## How NGINX Plus Validates a JWT A JWT is considered to be valid when the following conditions are met: - The signature can be verified (for JWS) or payload can be decrypted (for JWE) with the key found in the [`auth_jwt_key_file`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_file) or [`auth_jwt_key_request`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) (matching on the `kid` ("key ID"), if present, and `alg` ("algorithm") header fields). - The JWT is presented inside the validity period, when defined by one or both of the `nbf` ("not before") and `exp` ("expires") claims. ## Creating a JSON Web Key File In order to validate the signature with a key or to decrypt data, a JSON Web Key (`key.jwk`) should be created. The file format is defined by [JSON Web Key specification](https://tools.ietf.org/html/rfc7517): ```json {"keys": [{ "k":"ZmFudGFzdGljand0", "kty":"oct", "kid":"0001" }] } ``` where: - the `k` field is the generated symmetric key (base64url-encoded) basing on a `secret` (`fantasticjwt` in the example). The secret can be generated with the following command: ```shell echo -n fantasticjwt | base64 | tr '+/' '-_' | tr -d '=' ZmFudGFzdGljand0 ``` - the `kty` field defines the key type as a symmetric key (octet sequence) - the `kid` (Key ID) field defines a serial number for this JSON Web Key ## Getting JWKs from Subrequest NGINX Plus can be configured to fetch JSON Web Keys from the remote location - usually an identity provider, especially when using OpenID Connect. The IdP URI where the subrequest will be sent to is configured with the [`auth_jwt_key_request`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request) directive: ```nginx http { #... server { listen 80; #... location / { auth_jwt "closed site"; auth_jwt_key_request /_jwks_uri; # Keys will be fetched by subrequest proxy_pass http://my_backend; } } } ``` The URI may refer to an internal location (`_jwks_uri`) so that the JSON Web Key Set can be cached ([`proxy_cache`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache) and [`proxy_cache_path`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path) directives) to avoid validation overhead. Turning on caching is recommended for high-load API gateways even if [JWT key caching](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_cache) is used as it will help to avoid overwhelming a key server with key requests when a JWT key cache expires. ```nginx http { proxy_cache_path /var/cache/nginx/jwk levels=1 keys_zone=jwk:1m max_size=10m; #... server { listen 80; #... location = /_jwks_uri { internal; proxy_method GET; proxy_cache jwk; # Cache responses proxy_cache_valid 200 12h; proxy_pass https://idp.example.com/oauth2/keys; # Obtain keys from here } } } ``` The full example of getting JWKs from a subrequest: ```nginx # proxy_cache_path /var/cache/nginx/jwk levels=1 keys_zone=jwk:1m max_size=10m; server { listen 80; # Use SSL/TLS in production location / { auth_jwt "closed site"; auth_jwt_key_cache 1h; auth_jwt_key_request /_jwks_uri; # Keys will be fetched by subrequest proxy_pass http://my_backend; } location = /_jwks_uri { internal; proxy_method GET; proxy_cache jwk; # Cache responses proxy_cache_valid 200 12h; proxy_pass https://idp.example.com/oauth2/keys; # Obtain keys from here } } ``` ## Arbitrary JWT Claims Validation During JWT verification, NGINX Plus automatically validates only `nbf` ("not before") and `exp` ("expires") claims. However, in some cases you need to set more conditions for a successful JWT validation, in particular when dealing with application-specific or protocol level claims. For example, OpenID Connect Core requires validation of `iss` ("issuer"), `aud` ("audience"), `sub` ("subject") claims for `ID` token. Additional conditions for JWT validation can be set as variables with the [`map`](https://nginx.org/en/docs/http/ngx_http_map_module.html) module and then evaluated with the [`auth_jwt_require`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_require) directive. In this scenario, we are verifying that: - the recipient of the token (audience) is our APIs (map rule 1) - the token was issued by a trusted identity provider (map rule 2) - scopes in APIs called on behalf of administrators (map rule 3) The values of three resulting variables are evaluated in the [`auth_jwt_require`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_require) directive, and if the value of each variable is `1`, the JWT will be accepted: ```nginx upstream api_server { server 10.0.0.1; server 10.0.0.2; } map $jwt_claim_aud $valid_app_id { #map rule 1: "~api\d.example.com" 1; #token issued only for target apps } map $jwt_claim_iss $valid_issuer { #map rule 2: "https://idp.example.com/sts" 1; #token issued by trusted CA } map $jwt_claim_scope $valid_scope { #map rule 3: "access_as_admin" 1; #access as admin only } server { listen 80; location /products/ { auth_jwt "API"; auth_jwt_key_file conf/api_secret.jwk; auth_jwt_require $valid_app_id $valid_issuer $valid_scope; proxy_pass http://api_server; } } ``` In some cases the [`auth_jwt_require`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_require) directive can be specified multiple times, for example, for the purpose of authentication and then for authorization. In case of an error, the `401` code will be displayed. Assigning the custom error code `403` to another `auth_jwt_require` directive makes ti possible to differentiate authentication and authorization usecases and handle corresponding failures appropriately: ```nginx location /products/ { auth_jwt "API"; auth_jwt_key_file conf/api_secret.jwk; auth_jwt_require $valid_app_id $valid_issuer $valid_scope; auth_jwt_require $valid_scope error=403; proxy_pass http://api_server; } ``` ## Nested JWT Extraction A Nested JWT is a JWS token enclosed into JWE. In a Nested JWT, the sensitive information from JWS is protected with extra encryption of JWE. Using Nested JWT may be preferable over JWE because: - in case of JWE, the target application/service needs to decrypt the token first, then verify the signature. Decrypt operation on the application side may be time and resource consuming. - in case of Nested JWT, as NGINX Plus resides in the same trusted network with the target application, there is no need for token encryption between NGINX Plus and the application. NGINX Plus decrypts the JWE, checks the enclosed JWS, and sends the Bearer Token to the application. This will offload JWE decryption from the application to NGINX Plus. - if your application doesn't support JWE, using Nested JWT enables full protection for JWS. To enable Nested tokens: 1. Specify the `nested` type of JWT with the [`auth_jwt_type`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_type) directive. ```nginx auth_jwt_type nested; ``` 2. Pass the decrypted payload (the [`$jwt_payload`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#var_jwt_payload) variable) to the application as the Bearer token value in the `Authorization` header: ```nginx proxy_set_header Authorization "Bearer $jwt_payload"; ``` This example sums up the previous steps into one configuration: ```nginx upstream api_server { server 10.0.0.1; server 10.0.0.2; } http { server { listen 80; auth_jwt "API"; auth_jwt_type nested; auth_jwt_key_file conf/api_secret.jwk; proxy_pass http://api_server; proxy_set_header Authorization "Bearer $jwt_payload"; } } ``` ## See Also - [Authenticating API Clients with JWT and NGINX Plus](https://www.nginx.com/blog/authenticating-api-clients-jwt-nginx-plus/) -------------------------------------------------------------------------------- title: "Single Sign-On with Keycloak" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/keycloak/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Keycloak as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Keycloak](https://www.keycloak.org/) as the Identity Provider (IdP), and NGINX Plus as the Relying Party, or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. Starting from NGINX Plus version R34, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure Keycloak {#keycloak-setup} 1. Log in to your Keycloak admin console, for example, `https:///admin/master/console/`. 2. In the left navigation, go to **Clients**, then 3. Select **Create client** and provide the following details: - Set **Client type** to **OpenID Connect**. - Enter a **Client ID**, for example, `nginx-demo-app`. You will need it later when configuring NGINX Plus. - Select **Next**. 4. In the **Capability Config** section: - Set **Client Authentication** to **On**. This sets the client type to **confidential**. - Select **Next**. 5. In the **Login Settings** section: Add a **Redirect URI**, for example: ```text https://demo.example.com/oidc_callback ``` Add a **Post Logout Redirect URI** to support RP-initiated logout, for example: ```text https://demo.example.com/post_logout/ ``` - Select **Save**. 6. In the **Credentials** tab, make note of the **Client Secret**. You will need it later when configuring NGINX Plus. ### Configure Logout and Front-Channel Single Logout (optional) {#keycloak-frontchannel-logout} Front-channel logout allows Keycloak to notify NGINX Plus when a user signs out of the realm or another application that participates in Single Logout (SLO). Keycloak sends a front-channel HTTP request (typically loaded in a hidden iframe) to a logout URL that you configure for the client. 1. In the Keycloak admin console, go to **Clients** and select your `nginx-demo-app` client. 2. On the **Settings** tab, scroll to the **Logout settings** section. 3. Configure front-channel logout for this client: - Set **Front channel logout** to **On**. - In **Front-channel Logout URL**, enter the NGINX Plus front-channel logout endpoint, for example: ```text https://demo.example.com/front_logout ``` - Enable **Front-channel logout session required**. With this option enabled, Keycloak includes both the session identifier (`sid`) and issuer (`iss`) parameters in the front-channel logout request that it sends to NGINX Plus, as defined by the OpenID Connect Front-Channel Logout specification. NGINX Plus uses these parameters to identify and clear the corresponding user session. ### Assign Users or Groups This step is optional, and is necessary if you need to restrict or organize user permissions. 1. In the **Roles** tab, add a **Client Role**, for example, `nginx-keycloak-role`. 2. Under **Users**, create a new user or select a user. 3. In **Role Mappings**, assign a role to the user within the `nginx-demo-app` client. ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, Keycloak publishes the `.well-known/openid-configuration` document at the following address: `https:///realms//.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https:///realms//.well-known/openid-configuration | jq ``` Where: - the `` is your Keycloak server address - the `` is your Keycloak realm name - the `/.well-known/openid-configuration` is the default address for Keycloak for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https:///realms/", "authorization_endpoint": "https:///realms//protocol/openid-connect/auth", "token_endpoint": "https:///realms//protocol/openid-connect/token", "jwks_uri": "https:///realms//protocol/openid-connect/certs", "userinfo_endpoint": "https:///realms//protocol/openid-connect/userinfo", "end_session_endpoint": "https:///realms//protocol/openid-connect/logout", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for Keycloak is `https:///realms/`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ## Set up NGINX Plus {#nginx-plus-setup} With Keycloak configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Relying Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [Keycloak Configuration](#keycloak-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the Keycloak provider named `keycloak` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider keycloak { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - your actual Keycloak **Client ID** obtained in [Keycloak Configuration](#keycloak-setup) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - (if not using PKCE) your **Client Secret** obtained in [Keycloak Configuration](#keycloak-setup) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - the **Issuer** URL obtained in [Keycloak Configuration](#keycloak-setup) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your Keycloak OIDC URL: `https:///realms/`. By default, NGINX Plus creates the metadata URL by appending the `/.well-known/openid-configuration` part to the Issuer URL. If your metadata URL is different, you can explicitly specify it with the [`config_url`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url) directive. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where Keycloak should redirect the user after a successful logout. This value **must also be configured** in the Keycloak client's Post Logout Redirect URIs. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user's ID token as a *hint* to Keycloak. This directive is **optional**, however, if it is omitted the Keycloak may display an extra confirmation page asking the user to approve the logout request. - The **frontchannel_logout_uri** directive defines the URI on NGINX Plus that receives OpenID Connect front-channel logout requests from Keycloak. This path must be an HTTPS endpoint and must match the **Front-channel Logout URL** configured for the client in Keycloak. When a front-channel logout request is sent (typically in a hidden iframe), Keycloak includes the session identifier and issuer (the `sid` and `iss` parameters, when **Front-channel logout session required** is enabled); the OIDC module uses these values to locate and clear the corresponding local session. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch `/protocol/openid-connect/userinfo` from the Keycloak and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when Keycloak's OpenID Connect discovery document advertises the `S256` code challenge method in the `code_challenge_methods_supported` field. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP metadata does not list `S256`. - The module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`), which remains the default for Keycloak. - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider keycloak { issuer https:///realms/; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; frontchannel_logout_uri /front_logout; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with Keycloak OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `keycloak` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#keycloak-setup-oidc-provider): ```nginx # ... location / { auth_oidc keycloak; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by Keycloak: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by Keycloak - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc keycloak; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nKeycloak sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider keycloak { # The 'issuer' typically matches your Keycloak realm's base URL: # For example: https:///realms/ issuer https:///realms/master; # Replace with your actual Keycloak client_id and secret client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Front-channel logout (OP‑initiated single sign-out) frontchannel_logout_uri /front_logout; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration # pkce on; # If the .well-known endpoint can't be derived automatically, # specify config_url: # config_url https:///realms/master/.well-known/openid-configuration; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Protect this location with Keycloak OIDC auth_oidc keycloak; # Forward OIDC claims as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nKeycloak sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open https://demo.example.com/ in a browser. You should be redirected to Keycloak's login page for your realm. 2. Enter valid Keycloak credentials for a user assigned to the `nginx-demo-app` client. Upon successful sign-in, Keycloak redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; Keycloak ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to Keycloak for a fresh sign‑in, proving the session has been terminated. ## Legacy njs-based Keycloak Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based Keycloak Guide](nginx/deployment-guides/single-sign-on/oidc-njs/keycloak.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added front-channel logout support (`frontchannel_logout_uri`), PKCE configuration (`pkce` directive), and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "UDP Health Checks" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/udp-health-check/index.md description: This chapter describes how to configure different types of health checks for UDP servers in a load‑balanced upstream server group. -------------------------------------------------------------------------------- NGINX Plus can continually test your upstream servers that handle UDP network traffic (DNS, RADIUS, syslog), avoid the servers that have failed, and gracefully add the recovered servers into the load‑balanced group. ## Prerequisites {#prereq} - You have [configured an upstream group of servers](nginx/admin-guide/load-balancer/tcp-udp-load-balancer.md) that handles UDP network traffic in the [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context, for example: ```nginx stream { #... upstream dns_upstream { server 192.168.136.130:53; server 192.168.136.131:53; server 192.168.136.132:53; } #... } ``` - You have configured a server that passes UDP datagrams to the upstream server group: ```nginx stream { #... server { listen 53 udp; proxy_pass dns_upstream; proxy_timeout 1s; proxy_responses 1; error_log logs/dns.log; } #... } ``` See [TCP and UDP Load Balancing](nginx/admin-guide/load-balancer/tcp-udp-load-balancer.md) for details. ## Passive UDP health checks {#hc_passive} NGINX Open Source or F5 NGINX Plus can mark the server as unavailable and stop sending UDP datagrams to it for some time if the server replies with an error or times out. The number of consecutive failed connection attempts within a certain time period is set with the [`max_fails`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_fails) parameter for an [`upstream server`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#server) (default value is `1`). The time period is set with the [`fail_timeout`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#fail_timeout) parameter (default value is `10` seconds). The parameter also sets the amount of time that NGINX considers the server unavailable after marking it so. So if a connection attempt times out or fails at least once in a 10‑second period, NGINX marks the server as unavailable for 10 seconds. The example shows how to set these parameters to 2 failures within 60 seconds: ```nginx upstream dns_upstream { server 192.168.136.130:53 fail_timeout=60s; server 192.168.136.131:53 fail_timeout=60s; } ``` ## Active UDP health checks {#hc_active} Active Health Checks allow testing a wider range of failure types and are available only for NGINX Plus. For example, instead of waiting for an actual TCP request from a DNS client to fail before marking the DNS server as down (as in passive health checks), NGINX Plus will send special health check requests to each upstream server and check for a response that satisfies certain conditions. If a connection to the server cannot be established, the health check fails, and the server is considered unhealthy. NGINX Plus does not proxy client connections to unhealthy servers. If more than one health check is defined, the failure of any check is enough to consider the corresponding upstream server unhealthy. To enable active health checks: 1. In the upstream group, specify a _shared memory zone_ with the [`zone`](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#zone) directive – a special area where the NGINX Plus worker processes share state information about counters and connections. In the `zone` directive, specify the zone _name_ (`dns_zone` in the example) and the zone _size_ (`64k` in the example): ```nginx stream { #... upstream dns_upstream { zone dns_zone 64k; server 192.168.136.130:53; server 192.168.136.131:53; server 192.168.136.132:53; } #... } ``` 2. In the `server` block that forwards traffic to the upstream group (via [`proxy_pass`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_pass)), specify the [`udp`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_udp) parameter to the `health_check` directive: ```nginx stream { #... server { listen 53 udp; proxy_pass dns_upstream; health_check udp; } #... } ``` A basic UDP health check assumes that NGINX Plus sends the “nginx health check” string to an upstream server and expects the absence of ICMP “Destination Unreachable” message in response. You can configure your own health check tests in the `match {}` block. See [The “match {}” Configuration Block](#hc_active_match) for details. ### Fine-Tuning UDP Health Checks {#hc_active_finetune} You can fine‑tune the health check by specifying the following parameters to the [`health_check`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive: - `interval`– How often (in seconds) NGINX Plus sends health check requests (default is `5` seconds) - `passes`– Number of consecutive health checks the server must respond to to be considered healthy (default is `1`) - `fails`– Number of consecutive health checks the server must fail to respond to to be considered unhealthy (default is `1`) ```nginx server { listen 53 udp; proxy_pass dns_upstream; health_check interval=20 passes=2 fails=2 udp; } ``` In the example, the time between UDP health checks is increased to 20 seconds, the server is considered unhealthy after 2 consecutive failed health checks, and the server needs to pass 2 consecutive checks to be considered healthy again. ### The “match {}” configuration block {#hc_active_match} A basic UDP health check assumes that NGINX Plus sends the “nginx health check” string to an upstream server and expects the absence of ICMP “Destination Unreachable” message in response. You can configure your own health check tests that will verify server responses. These tests are defined within the [`match {}`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) configuration block. 1. In the top‑level `stream {}` context, specify the [`match {}`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) block and set its name, for example, `udp_test`: ```nginx stream { #... match udp_test { #... } } ``` 2. Refer to the block from the `health_check` directive by including the [`match`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_match) parameter to specify the name of the `match {}` block: ```nginx stream { #... server { listen 53 udp; proxy_pass dns_upstream; health_check match=udp_test udp; } #... } ``` 3. In the `match {}` block, specify conditions or tests under which a health check succeeds. This is done with `send` and `expect` parameters: - [`send`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match_send)– The text string or hexadecimal literals (“/x” followed by two hex digits) to send to the server - [`expect`](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match_expect)– Literal string or regular expression that the data returned by the server needs to match These parameters can be used in different combinations, but no more than one `send` and one `expect` parameter can be specified at a time. ## Usage scenarios ### NTP health checks {#example_ntp} To fine‑tune health checks for NTP, you should specify both `send` and `expect` parameters with the following text strings: ```nginx match ntp { send \xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00; expect ~* \x24; } ``` #### Complete NTP health check configuration example ```nginx stream { upstream ntp_upstream { zone ntp_zone 64k; server 192.168.136.130:53; server 192.168.136.131:53; server 192.168.136.132:53; } server { listen 53 udp; proxy_pass ntp_upstream; health_check match=ntp udp; proxy_timeout 1s; proxy_responses 1; error_log logs/ntp.log; } match ntp { send \xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00; expect ~* \x24; } } ``` ### DNS health checks {#example_dns} [DNS health checks](#hc_active) can be enhanced to perform real DNS lookup queries. You can craft a valid DNS query packet, send it to the upstream server, and inspect the response to determine health. The process includes three steps: - [Creating a CNAME test record](#create-a-cname-record) on your DNS server. - [Crafting a raw DNS query packet](#construct-a-raw-dns-query-packet) to be sent by NGINX Plus. - [Validating the expected response](#configure-the-match-block-for-dns-lookup) using the `match` block, where the `send` parameter represents a raw DNS query packet, and `expect` represents the value of the CNAME record. #### Create a CNAME record First, create a CNAME record on your DNS server for a health check that points to the target website. For example, if you are using BIND self-hosted DNS solution on a Linux server: - Open the zone file in a text editor: ```shell sudo nano /etc/bind/zones/db.example.com ``` - Add a CNAME record, making `healthcheck.example.com` resolve to `healthy.svcs.example.com`: ```none healthcheck IN CNAME healthy.svcs.example.com. ``` - Save the file and reload the DNS service: ```shell sudo systemctl reload bind9 ``` Once the CNAME record is live and resolvable, you can craft a DNS query packet that represents a DNS lookup and can be used in the `send` directive. #### Construct a raw DNS query packet The `send` parameter of the `match` block allows you to send raw UDP packets for health checks. To query your CNAME record, you need to construct a valid DNS query packet according to the [DNS protocol message format](https://datatracker.ietf.org/doc/html/rfc1035#section-4.1), including a header and question section. The DNS Query packet can be created using DNS packet builders, such as Python Scapy or dnslib, or packet analyzers, such as tcpdump or Wireshark. If using a packet analyzer, extract only the DNS layer, removing Ethernet, IP, and UDP-related headers. This is the raw DNS query of `healthcheck.example.com`, represented as one line in Hex with `\x` prefixes: ```none \x00\x01\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x0b\x68\x65\x61\x6c\x74\x68\x63\x68\x65\x63\x6b\x07\x65\x78\x61\x6d\x70\x6c\x65\x03\x63\x6f\x6d\x00\x00\x01\x00\x01 ``` where: | HEX | Description | |------------------|------------------------| | \x00\x01 | Transaction ID: 0x0001 | | \x01\x00 | Flags: Standard query, recursion desired | | \x00\x01 | Questions: 1 | | \x00\x00 | Answer RRs: 0 | | \x00\x00 | Authority RRs: 0 | | \x00\x00 | Additional RRs: 0 | | \x0b\x68\x65\x61\x6c\x74\x68\x63\x68\x65\x63\x6b | "healthcheck" | | \x07\x65\x78\x61\x6d\x70\x6c\x65 | "example" | | \x03\x63\x6f\x6d | "com" | | \x00 | end of name | | \x00\x01 | Type: A | | \x00\x01 | Class: IN | #### Configure the match block for DNS lookup Finally, specify the `match` block in the NGINX configuration file to pair the raw query with an expected response. The `send` directive should contain the DNS query packet, while `expect` directive should contain a matching DNS record in the DNS server's response: ```nginx match dns { send \x00\x01\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x0b\x68\x65\x61\x6c\x74\x68\x63\x68\x65\x63\x6b\x07\x65\x78\x61\x6d\x70\x6c\x65\x03\x63\x6f\x6d\x00\x00\x01\x00\x01; expect ~* "healthy.svcs.example.com"; } ``` #### Complete DNS health check configuration example ```nginx stream { upstream dns_upstream { zone dns_zone 64k; server 192.168.136.130:53; server 192.168.136.131:53; server 192.168.136.132:53; } server { listen 53 udp; proxy_pass dns_upstream; health_check match=dns udp; proxy_timeout 1s; proxy_responses 1; error_log logs/dns.log; } match dns { # make sure appropriate CNAME record exists send \x00\x01\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x0b\x68\x65\x61\x6c\x74\x68\x63\x68\x65\x63\x6b\x07\x65\x78\x61\x6d\x70\x6c\x65\x03\x63\x6f\x6d\x00\x00\x01\x00\x01; expect ~* "healthy.svcs.example.com"; } } ``` ## See also - [Load Balancing DNS Traffic with NGINX and NGINX Plus](https://www.f5.com/company/blog/nginx/load-balancing-dns-traffic-nginx-plus) - [TCP/UDP Load Balancing with NGINX: Overview, Tips, and Tricks](https://blog.nginx.org/blog/tcp-load-balancing-udp-load-balancing-nginx-tips-tricks#activeHealthCheck) -------------------------------------------------------------------------------- title: "Using NGINX and NGINX Plus as an Application Gateway with uWSGI and Django" url: https://docs.nginx.com/nginx/admin-guide/web-server/app-gateway-uwsgi-django/index.md description: Configure NGINX and F5 NGINX Plus as an application gateway for uWSGI and Django -------------------------------------------------------------------------------- This article explains how to use NGINX or F5 NGINX Plus as an application gateway with uWSGI and Django. ## Introduction NGINX is a high‑performance, scalable, secure, and reliable web server and a reverse proxy. NGINX uses web acceleration techniques to manage HTTP connections and traffic. Features like [load balancing](../load-balancer/http-load-balancer.md), [SSL termination](/nginx/admin-guide/security-controls/terminating-ssl-http.md), connection and request [policing](/nginx/admin-guide/security-controls/controlling-access-proxied-http.md), static [content offload](/nginx/admin-guide/web-server/serving-static-content.md), and [content caching](/nginx/admin-guide/content-cache/content-caching.md) help users build reliable websites. NGINX acts as a secure application gateway, passing traffic from users to applications. In this regard, not only can NGINX proxy HTTP and HTTPS traffic to an HTTP‑enabled application container, it can also connect to most of the popular application servers and web frameworks via optimized app‑gateway interfaces implemented in modules like [FastCGI](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html), [Memcached](https://nginx.org/en/docs/http/ngx_http_memcached_module.html), [scgi](https://nginx.org/en/docs/http/ngx_http_scgi_module.html), and [uwsgi](https://nginx.org/en/docs/http/ngx_http_uwsgi_module.html). Many application containers have embedded external HTTP interfaces with some routing capabilities. NGINX offers an all‑in‑one solution. It handles HTTP connection management, load balancing, content caching, and traffic security. The application backend sits behind NGINX for better scalability and performance. You can group app instances behind NGINX to ensure high availability. ## About uWSGI and Django A few words about "specialized interfaces." As useful as it is, HTTP has never been designed for modern, lightweight application‑deployment scenarios. Over time, standardized interfaces have evolved for use with various application frameworks and application containers. One of these interfaces is the Web Server Gateway Interface ([WSGI](http://wsgi.readthedocs.org/en/latest/)), an interface between a web server/proxy and Python‑based applications. One common application server is the [uWSGI application server container](https://github.com/unbit/uwsgi). It offers [uwsgi](http://uwsgi-docs.readthedocs.org/en/latest/Protocol.html) - its own implementation of the WSGI protocol. Other than that, the uWSGI application server supports HTTP, FastCGI, and SCGI – with the uwsgi protocol recommended as the fastest way to talk to applications. ## Configure NGINX and NGINX Plus for Use with uWSGI and Django This document provides an example of how to configure NGINX and NGINX Plus for use with a [uWSGI](http://uwsgi-docs.readthedocs.org/en/latest/) server and a Python development environment. NGINX 0.8.40 and later (and all releases of NGINX Plus) includes native support for passing traffic from users to Python applications via the uwsgi protocol. If you downloaded [NGINX Open Source binaries or source](https://nginx.org/en/download.html) from our official repositories, or [NGINX Plus from the customer portal](https://account.f5.com/myf5), no action is needed to enable support for the uwsgi protocol – NGINX and NGINX Plus support uswgi by default. Configuring the uWSGI application container itself is outside the scope of this document; refer to the excellent [Quickstart for Python/WSGI applications](http://uwsgi-docs.readthedocs.org/en/latest/WSGIquickstart.html) for more information. [Django](https://www.djangoproject.com/) is a common Python web framework. For simplicity the example uses a Django‑based setup for the Python app. The [Django documentation](https://docs.djangoproject.com/en/1.11/) provides extensive information on how to configure a Django environment. This example is illustrative only, and one way you might invoke your uWSGI server with Django: ```shell uwsgi \ --chdir=/var/django/projects/myapp \ --module=myapp.wsgi:application \ --env DJANGO_SETTINGS_MODULE=myapp.settings \ --master --pidfile=/usr/local/var/run/uwsgi/project-master.pid \ --socket=127.0.0.1:29000 \ --processes=5 \ --uid=505 --gid=505 \ --harakiri=20 \ --max-requests=5000 \ --vacuum \ --daemonize=/usr/local/var/log/uwsgi/myapp.log ``` With these options in place, here's a sample NGINX configuration for use with a Django project: ```nginx http { # ... upstream django { server 127.0.0.1:29000; } server { listen 80; server_name myapp.example.com; root /var/www/myapp/html; location / { index index.html; } location /static/ { alias /var/django/projects/myapp/static/; } location /main { include /etc/nginx/uwsgi_params; uwsgi_pass django; uwsgi_param Host $host; uwsgi_param X-Real-IP $remote_addr; uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for; uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto; } } } ``` This configuration defines an upstream named `django`. The port number `29000` specified for the server in this upstream matches the port that the uWSGI server binds to, as specified by the `--socket=` argument in the sample `uwsgi` command. Serving static content is offloaded to NGINX or NGINX Plus, which serves it directly from `/var/django/projects/myapp/static`. Application traffic to the `/main` location is proxied and bridged from HTTP to the uwsgi protocol, and then passed to the Django app that runs within the uWSGI application container. ## Conclusion Lightweight, heterogeneous application environments are becoming a popular way of building and deploying modern web applications. Newer, standardized application interface protocols like uwsgi and FastCGI enable faster communication between users and applications. Using NGINX and NGINX Plus in front of an application container has become a common way to free applications from the burden of HTTP traffic management, and to protect the application from unexpected spikes of user traffic, malicious behavior, denial‑of‑service (DoS) attacks, and more. This allows developers to fully focus on the application logic, and leave the web acceleration and fundamental HTTP traffic security tasks to NGINX or NGINX Plus. ## Resources - [NGINX support](https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html) in the uWSGI project documentation - [How to use Django with uWSGI](https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/uwsgi/) in the Django project documentation -------------------------------------------------------------------------------- title: "Single Sign-On with OpenID Connect and Identity Providers" url: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-oidc/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using an Identity Provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus using: - OpenID Connect as the authentication mechanism. - An external Identity Provider (IdP) such as AD FS, Auth0, Cognito, Entra ID, Keycloak, OneLogin, Okta, Ping Identity and others. - NGINX Plus as an OIDC client application that verifies user identity (Relying Party). OpenID Connect is an identity protocol that utilizes the authorization and authentication mechanisms of OAuth 2.0. With it, NGINX Plus can provide a layer of authentication for protected applications that do not natively support it. NGINX Plus consumes authorization and claims from OpenID Connect Identity Providers by utilizing JWT-based identity tokens that are delivered via the OAuth 2.0 framework. It supports several specific flows suitable for browser-based and desktop/mobile applications. OpenID Connect allows the client (NGINX Plus) to retrieve an ID token in addition to an access token. The ID token provides information about the authenticated user. For the target client application, OIDC authentication can be enabled with great flexibility on different levels. It can be enabled globally, or more granular at a per-server or a per-location level. Additionally, OIDC supports auto discovery and retrieval of the OpenID provider configuration metadata while also allowing the definition of additional metadata if needed. ## OpenID Connect Workflow and NGINX Plus {#oidc-workflow} 1. A user accesses a protected resource. 2. NGINX Plus redirects the user to the IdP for user authentication and authorization. 3. The IdP collects user credentials and authenticates the user. 4. The IdP redirects the user back to NGINX Plus with an authorization code. 5. NGINX Plus retrieves an `id_token` and access token using the authorization code from the IdP. 6. NGINX Plus validates the `id_token` and retrieves profile data for the user using the `UserInfo` endpoint. The retrieved profile data is validated and the content of the `id_token` and the profile data is used for providing access control to the client application. 7. Upon successful validation, the resource access request is sent to the client application along with the access token. 8. The client application validates the access token and based on the token validation, the resource access request is allowed or denied. ![image](nginx/images/oidc.png) ## Tested Identity Providers {#deployment-guildes} NGINX Plus has tested support with the following Identity Providers: | IdP Provider | Resource | |-----------------|-------------------------------------------------------------------------------------------------------------| | Amazon Cognito | [Deployment Guide for Amazon Cognito](nginx/deployment-guides/single-sign-on/cognito.md) | | Auth0 | [Deployment Guide for Auth0](nginx/deployment-guides/single-sign-on/auth0.md) | | Microsoft AD FS | [Deployment Guide for AD FS](nginx/deployment-guides/single-sign-on/active-directory-federation-services.md) | | Microsoft Entra / Azure ID| [Deployment Guide for Entra ID](nginx/deployment-guides/single-sign-on/entra-id.md) | | Keycloak | [Deployment Guide for Keycloak](nginx/deployment-guides/single-sign-on/keycloak.md) | | OneLogin | [Deployment Guide for OneLogin](nginx/deployment-guides/single-sign-on/onelogin.md) | | Okta | [Deployment Guide for Okta](nginx/deployment-guides/single-sign-on/okta.md) | | Ping Identity | [Deployment Guide for Ping Identity](nginx/deployment-guides/single-sign-on/ping-identity.md) | ## Prerequisites {#prerequisites} - An identity provider (IdP) set up on your network or in the cloud. You need admin access to the IdP. - An [NGINX Plus subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 34](nginx/releases.md#r34) or later. To install NGINX Plus, follow the steps in [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name that points to your NGINX Plus instance (for example, `demo.example.com`). ## Set up your identity provider (IdP) {#idp-setup} The setup steps are similar for most identity providers, but some details may differ. 1. Log in to your IdP's admin console. 2. Create a new OpenID Connect (OIDC) application. - Give the app a name. - Add the users or groups who need access. 3. Find the **Client ID** and **Client Secret** for your app. You'll need these later when you [set up NGINX Plus as the relying party](#setup-oidc-provider2). 4. Find the **issuer** value. You can find the issuer value in your IdP app settings or at the standard discovery URL: `https://your-idp-domain/.well-known/openid-configuration` where: - the `your-idp-domain` is your IdP's server address - the `.well-known/openid-configuration` is the default address for IdPs for the `.well-known` document - the `userinfo_endpoint` is URI that a user visits to start an RP‑initiated logout flow. - the `end_session_endpoint` is the URL where your IdP should redirect the user after a successful logout. The IdP configuration metadata is returned in the JSON format, for example: ```json { ... "issuer": "https://your-idp-domain/idp", "authorization_endpoint": "https://your-idp-domain/idp/oauth2/authorize/", "token_endpoint": "https://your-idp-domain/idp/oauth2/token/", "jwks_uri": "https://your-idp-domain/idp/discovery/keys", "userinfo_endpoint": "https://your-idp-domain/idp/userinfo", "end_session_endpoint": "https://your-idp-domain/idp/oauth2/logout", ... } ``` Copy the **issuer** value. You will need it later when [configuring NGINX Plus as the Relying Party](#setup-oidc-provider2). 5. Configure a logout URI - a URI that a user visits to initiate an RP‑initiated logout flow, for example, `https://demo.example.com/post_logout/`. (Optional, supported since [NGINX Plus R35](nginx/releases.md#r35)). ## Configure the Relying Party (NGINX Plus) {#rp-setup} With your IdP configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 35: ```none nginx version: nginx/1.29.0 (nginx-plus-r35) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained from your IdP Provider. 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the IdP provider named `my_idp` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider my_idp { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - Your actual **Client ID** obtained from your IdP with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - Your **Client Secret** obtained from your IdP with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - The **Issuer** URL obtained from your IdP with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive By default, NGINX Plus creates the metadata URL by appending the `/.well-known/openid-configuration` part to the Issuer URL. If your Issuer is different, you can explicitly specify the metadata document with the [`config_url`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url) directive. - A valid system CA bundle with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) so that NGINX Plus could validate the IdP TLS certificates: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider my_idp { issuer https://your-idp-domain/idp; client_id ; client_secret ; ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with IdP OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `my_idp` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#setup-oidc-provider): ```nginx # ... location / { auth_oidc my_idp; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by the IdP: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by the IdP - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - Any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc my_idp; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nIdP sub sub: $http_sub\n"; default_type text/plain; } } ``` 11. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example {#example} This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider my_idp { # The 'issuer' typically matches your IdP's base URL issuer https:///idp; # Provide a CA bundle for certificate validation ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt; # Replace with your actual IdP's client_id and secret client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Fetch userinfo claims userinfo on; # If the .well-known endpoint cannot be derived automatically, # specify config_url: # config_url https:///auth/realms/main/.well-known/openid-configuration; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Protect this location with OIDC auth_oidc my_idp; # Forward OIDC claims as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # simple test oidc-protected application listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nIdP sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing {#testing} 1. Open https://demo.example.com/ in a browser. You should be redirected to your IdP's login page. 2. Enter valid IdP credentials for a user assigned to the `nginx-demo-app` client. Upon successful sign-in, the IdP redirects you back to NGINX Plus, and you will see the proxied application content (for example, “Hello, Jane Doe!”). ## Glossary {#glossary} | Term | Description | |-------------------------|----------------------------------------------------| | Identity Provider (IdP) | A service that authenticates users and verifies their identity for client applications. | | Protected Resource | A resource that is hosted by the resource server and requires an access token to be accessed. | | Relying Party (RP) | A client service required to verify user identity. | | JSON Web Token (JWT) | An open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. | | ID Token | Specific to OIDC, the primary use of the token in JWT format is to provide information about the authentication operation's outcome. | | Access Token | Defined in OAuth2, this (optional) short lifetime token provides access to specific user resources as defined in the scope values in the request to the authorization server (can be a JSON token as well). | | Refresh Token | Coming from OAuth2 specs, the token is usually long-lived and may be used to obtain new access tokens. | ## See Also {#see-also} - [Simplifying OIDC and SSO with the New NGINX Plus R34 OIDC Module blog post](https://community.f5.com/kb/technicalarticles/simplifying-oidc-and-sso-with-the-new-nginx-plus-r34-oidc-module/340552) - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R34](nginx/releases.md#r34) -------------------------------------------------------------------------------- title: "Create Amazon EC2 Instances for NGINX Open Source and NGINX Plus" url: https://docs.nginx.com/nginx/deployment-guides/amazon-web-services/ec2-instances-for-nginx/index.md description: Create Amazon Elastic Compute Cloud (EC2) instances for running NGINX Open Source and F5 NGINX Plus. -------------------------------------------------------------------------------- This guide explains how to create instances in the Amazon Elastic Compute Cloud (EC2) that can run NGINX Open Source and F5 NGINX Plus. For NGINX Plus, you can buy a prebuilt Amazon Machine Image (AMI) from the AWS Marketplace for a faster option. You can find AMIs for various operating systems, such as Amazon Linux, Red Hat Enterprise Linux, and Ubuntu. For instructions, see [Installing NGINX Plus AMIs on Amazon EC2](/nginx/admin-guide/installing-nginx/installing-nginx-plus-amazon-web-services.md). ## Before you begin To complete this guide, you need the following: - An [AWS account](http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/AboutAWSAccounts.html). - Basic Linux system administration skills, including installing software, managing files and folders, and using the command line, to follow the [Ansible instructions](#automate-ansible). - A paid or trial subscription for NGINX Plus. - Root privileges on the hosts where NGINX Open Source or NGINX Plus will be installed, with `sudo` access as needed. ## Create an Amazon EC2 Instance {#create-ec2-instances} 1. Log in to the EC2 dashboard in the AWS Management Console: **[https://console.aws.amazon.com/ec2](https://console.aws.amazon.com/ec2)**. 1. In the left navigation bar, choose **Instances**. Then select **Launch Instances** in the top right corner. 1. On the **Launch an Instance** page, give your new instance a name in the **Name and Tags** section. This name will show in the Name column of the summary table on the EC2 Instances dashboard. This guide is using "instance-name." 1. In the **Application and OS Images (Amazon Machine Image)** section select the image of the Linux distribution of your choice. ![Screenshot of AMI section on the EC2 Launch Instance page](/img/aws/aws-nlb-instance-choose-ami.png) 1. In the **Instance Type** section, choose an appropriate instance type. The screenshot shows the **t2.micro** instance type selected by default. This type is sufficient for demo purposes. **Note:** At the time of publication, AWS offered 750 hours of free usage each month for this instance type. This applies during your first year with an AWS account. Keep in mind, though, that several NGINX instances running all day will use the free 750 hours up quickly. For example, 6 instances will use them in just over 5 days. If you use 8 instances, you'll hit the limit in under 4 days. ![Screenshot of Instance Type on the EC2 Launch Instance page](/img/aws/aws-nlb-instance-choose-type.png) 1. In the **Key pair (login)** section, select an existing key pair or create a new one. If you choose **Create new key pair**, a window appears, allowing you to download the key pair. **Note:** It's best practice — and necessary in production — to create a separate key for each EC2 instance. This way, if a key is compromised, only that one instance is at risk. 1. Scroll to the **Network settings** section. You can keep the default **VPC** and **Subnet** settings. Under **Firewall (Security Groups),** either create a new security group or use an existing one. - If this is your first setup, keep **Create security group** selected. - Select **Allow HTTP traffic from the internet.** - (Optional) Select **Allow HTTPS traffic from the internet** if needed. This creates the following inbound security group rules: 1. Accept SSH connections from all sources - **Type** – SSH - **Protocol** – TCP - **Port Range** – 22 - **Source** – Custom `0.0.0.0/0` 1. Accept unencrypted HTTP connections from all sources - **Type** – HTTP - **Protocol** – TCP - **Port Range** – 80 - **Source** – Custom `0.0.0.0/0` 1. Accept encrypted HTTPS connections from all soruces (optional) - **Type** – HTTPS - **Protocol** – TCP - **Port Range** – 443 - **Source** – Custom `0.0.0.0/0` ![Screenshot of Network Settings on the EC2 Launch Instance page](/img/aws/aws-generic-instance-details.png) If you are deploying multiple instances, it makes sense to use the same security group for all. In that case, choose "Select existing security group." This will allow you to pick a security group from a list. ![Screenshot of Network Settings on the EC2 Launch Instance page with the option of selecting an existing security group](/img/aws/aws-instance-select-sg.png) 1. In the **Configure Storage** section, leave the defaults unchanged. 1. In the right panel, verify the settings in the **Summary** section. If everything is correct, select **Launch Instance**. ![Screenshot of the Summary panel on the EC2 Launch Instance page with button to launch instance](/img/aws/aws-instance-summary.png) 1. After launching the instance, you are redirected to a confirmation page with a success message. Select the instance ID to return to the **EC2 Instances** page. ![Screenshot of the Summary panel on the EC2 Launch Instance page with button to launch instance](/img/aws/aws-instance-launch-success.png) 1. On the **EC2 Instances** page, you can view all your instances, including the new one. The following screenshot shows a single instance: ![Screenshot of the EC2 Instances page with a single instance](/img/aws/aws-generic-instance-display-first.png) 1. Finalize your security group rules. You only need to do this for the first instance in a set. All other instances in that set can use the same security group. - In the left navigation bar, select **Security Groups**. - Select the security group by clicking its radio button in the leftmost column of the table. A panel opens in the lower part of the window displaying details about the group. - In the **Inbound** tab, verify that the rules you created in Step 7 are listed. - Open the **Outbound** tab and select **Edit outbound rules** to create a rule for outbound traffic. The rules depend on the ports used for traffic handled by NGINX Open Source or NGINX Plus instances: - By default, AWS adds an outbound rule that allows all traffic to all destinations. You can remove this rule by selecting **Delete**. - If you use port 80 for client traffic and health checks from a load balancer, like [AWS Network Load Balancer](high-availability-network-load-balancer.md), you only need one rule. - If you set up different ports for various tasks, or if you use ports like 443 for HTTPS, adjust them accordingly. In the **Destination** field, start typing your security group's name or ID. It should appear under **Security Groups**. The example below shows **sg-0dd4d3c5284052f99**. ![Screenshot of the EC2 Security Group outbound rules page](/img/aws/aws-generic-instance-security-outbound.png) 1. To install NGINX software on the instance, first [connect](#connect-to-an-ec2-instance) to it. Then follow the instructions in the NGINX Plus Admin Guide for [NGINX Open Source](/nginx/admin-guide/installing-nginx/installing-nginx-open-source#prebuilt) and [NGINX Plus](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md). --- ## Connect to an EC2 Instance To install and configure NGINX Open Source or NGINX Plus on an instance, open a terminal window and connect to the instance over SSH. 1. Navigate to the **Instances** tab on the EC2 Dashboard if you are not there already. 1. Select the row for an instance to highlight it. 1. Select **Connect** above the list of instances. You are redirected to the **Connect to Instance** page, with the **SSH client** tab selected by default. 1. Follow the instructions on the page, which are customized for the selected instance. A sample `ssh` command includes the key file name and the instance hostname. ![Screenshot of the EC2 Instance Connect with SSH page](/img/aws/aws-nlb-instance-connect.png) --- ## Install NGINX software Once you have established a connection with an instance, you can install the NGINX software on it. Follow the instructions in the NGINX Plus Admin Guide for [NGINX Open Source](/nginx/admin-guide/installing-nginx/installing-nginx-open-source#prebuilt) and [NGINX Plus](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md). The [Admin Guide](/nginx/admin-guide/_index.md) also provides instructions for many maintenance tasks. ### Automate installation with a configuration manager You can automate the installation of NGINX Open Source and NGINX Plus. Instructions for Ansible are provided below. #### Automate installation with Ansible {#automate-ansible} NGINX, Inc. releases a combined Ansible role for NGINX Open Source and NGINX Plus on [Ansible Galaxy](https://galaxy.ansible.com/nginxinc/nginx/) and [GitHub](https://github.com/nginxinc/ansible-role-nginx). Perform these steps to install and run it. 1. [Connect to the EC2 instance](#connect-instance). 1. Install Ansible following the [instructions](https://docs.ansible.com/ansible/latest/installation_guide/installation_distros.html) for the operating system on your EC2 instance. 1. Install the official Ansible role from NGINX: ```shell ansible-galaxy install nginxinc.nginx ``` 1. (NGINX Plus only) Copy the **nginx-repo.key** and **nginx-repo.crt** files provided by NGINX, Inc. to **~/.ssh/ngx-certs/**. 1. Create a file called **playbook.yml** with the following contents: ```none --- - hosts: localhost become: true roles: - role: nginxinc.nginx ``` 1. Run the playbook: ```shell ansible-playbook playbook.yml ``` 1. Confirm that NGINX is installed by running `nginx -v`. ## Optional: Create an NGINX Open Source AMI To simplify installing NGINX Open Source on several instances, create an AMI from an existing NGINX instance. You can then create additional virtual instances based on the AMI. 1. Follow the instructions in [Create Amazon EC2 Instance](#create-ec2-instances) and [Install NGINX software](#install-nginx-software). 1. Go to the **Instances** tab on the Amazon EC2 Dashboard. 1. Select the base instance with NGINX installed by clicking its row in the table. 1. Select **Actions**, then choose **Image and templates** > **Create Image**. ![Screenshot of the EC2 Create Image button menu](/img/aws/aws-generic-create-image-menu.png) 1. On the **Create Image** page, enter the **Image name** and optionally add an **Image description**. Then select **Create image**. You are returned to the **Instances** page with a green alert confirming that the image is being created. 1. To verify that the image was created, go to the **AMIs** tab and find the new image by its name. ### Revision History - Version 3 (February 2025) - Update guide to match new AWS instance creation flow, update screenshots, and links. - Version 2 (July 2018) – Substitute links to NGINX Plus Admin Guide for sample installation instructions. - Version 1 (April 2018) – Initial version (NGINX Plus Release 14) -------------------------------------------------------------------------------- title: "Deploying NGINX and NGINX Plus with Docker" url: https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-docker/index.md description: Deploy NGINX and F5 NGINX Plus as the Docker container. -------------------------------------------------------------------------------- [F5 NGINX Plus](https://www.nginx.com/products/nginx/), the high‑performance application delivery platform, load balancer, and web server, is available as the Docker container. ## Prerequisites - The [Docker Engine](https://docs.docker.com/engine/install/) command-line tool - for NGINX Plus: * the JSON Web Token (JWT) from [MyF5](https://my.f5.com) customer portal * the *nginx-repo.crt* and *nginx-repo.key* files from [MyF5](https://my.f5.com) customer portal * your private Docker registry is configured and running - for NGINX Open Source: [Docker Hub](https://hub.docker.com/) account ## Use official NGINX Plus Docker images Since NGINX Plus [Release 31](nginx/releases.md#r31) you can get an NGINX Plus image from the official NGINX Plus Docker registry and upload it to your private registry. The NGINX Plus Docker registry is available at `https://private-registry.nginx.com/v2/`. The registry contains the following image types: - [NGINX Plus](/nginx/releases.md):
`https://private-registry.nginx.com/v2/nginx-plus/base` - [Unprivileged](/nginx/admin-guide/installing-nginx/installing-nginx-plus.md#unpriv_install) installation of NGINX Plus:
`https://private-registry.nginx.com/v2/nginx-plus/rootless-base` - NGINX Plus bundled with [NGINX Agent](https://docs.nginx.com/nginx-agent/overview/):
`https://private-registry.nginx.com/v2/nginx-plus/agent` - Unprivileged installation of NGINX Plus and NGINX Agent:
`https://private-registry.nginx.com/v2/nginx-plus/rootless-agent` - [NGINX Plus dynamic modules](/nginx/admin-guide/dynamic-modules/dynamic-modules.md):
`https://private-registry.nginx.com/v2/nginx-plus/modules` The images can be targeted for a particular operating system and NGINX Plus release using tags. ### Tags for operating systems | Operating system | Basic OS tag | Tag examples | |--------------------------------------------|--------------|-----------------------------------------------| | Alpine (x86_64, aarch64) | `alpine` | `r36-alpine`, `r36-alpine-3.20` | | Debian (x86_64, aarch64) | `debian` | `r36-debian`, `r36-debian-bookworm` | | Red Hat Enterprise Linux (x86_64, aarch64) | `ubi` | `r36-ubi`, `r36-ubi-9`, `r36-ubi-9-20251201` | ### Tags for NGINX Plus versions The NGINX Plus registry contains images for the two most recent versions of NGINX Plus. The basic operating system tag returns the latest version of NGINX Plus built for the latest version of this operating system. **Note:** `nginx-plus-r36-ubi-9`, `nginx-plus-r36-alpine-3.22`. The image may contain a particular version of NGINX Plus or contain a bundle of NGINX Plus and NGINX Agent, and can be targeted for a specific architecture. ### List all tags For a complete tag list for NGINX Plus bundled with NGINX Agent images, use the command: ```shell curl https://private-registry.nginx.com/v2/nginx-plus//tags/list --key --cert | jq ``` where: - the `` is the location of images in NGINX Plus private registry: - `base` is NGINX Plus only - `rootless-base` is NGINX Plus run from `nginx` user - `agent` is NGINX Plus along with NGINX Agent in a single image - `rootless-agent` is NGINX Plus with NGINX Agent both run from `nginx` user - `modules` is NGINX Plus dynamic modules - the `` is a local path to your client key from MyF5, for example, `/etc/ssl/nginx/nginx-repo-x12345.key` - the `` is a local path to your client certificate from MyF5, for example,`/etc/ssl/nginx/nginx-repo-x12345.crt` - the `jq` command is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. ### Download your subscription credential files In order to obtain a container image, you will need the JSON Web Token file or SSL certificate and private key files provided with your NGINX Plus subscription. These files grant access to the package repository from which the script will download the NGINX Plus package: #### JSON Web Token #### SSL ### Set up Docker for the F5 Container Registry This step describes how to use Docker to communicate with the F5 Container Registry located at `private-registry.nginx.com`. **Note:** The steps provided are for Linux. For Mac or Windows, see the [Docker for Mac](https://docs.docker.com/docker-for-mac/#add-client-certificates) or [Docker for Windows](https://docs.docker.com/docker-for-windows/#how-do-i-add-client-certificates) documentation. For more details on Docker Engine security, you can refer to the [Docker Engine Security documentation](https://docs.docker.com/engine/security/). #### JSON Web Token Open the JSON Web Token file previously downloaded from [MyF5](https://my.f5.com) customer portal (for example, `nginx-repo-12345abc.jwt`) and copy its contents. Log in to the Docker registry using the contents of the JSON Web Token file: ```shell docker login private-registry.nginx.com --username= --password=none ``` #### SSL Create a directory and copy your certificate and key to this directory: ```shell mkdir -p /etc/docker/certs.d/private-registry.nginx.com cp /etc/docker/certs.d/private-registry.nginx.com/client.cert cp /etc/docker/certs.d/private-registry.nginx.com/client.key ``` ### Pull the image Next, pull the image you need from `private-registry.nginx.com`. To pull an image, replace `` with the specific NGINX Plus version or the NGINX Plus version and OS version you need. For NGINX Plus, run: ```shell docker pull private-registry.nginx.com/nginx-plus/base: ``` For NGINX Plus with NGINX Agent version 2, run: ```shell docker pull private-registry.nginx.com/nginx-plus/agent: ``` For NGINX Plus with NGINX Agent version 3, run: ```shell docker pull private-registry.nginx.com/nginx-plus/agentv3: ``` For NGINX Plus installed from `nginx` user (rootless installation), run: ```shell docker pull private-registry.nginx.com/nginx-plus/rootless-base: ``` For NGINX Plus with NGINX Agent version 2 installed from `nginx` user (rootless installation), run: ```shell docker pull private-registry.nginx.com/nginx-plus/rootless-agent: ``` For NGINX Plus with NGINX Agent version 3 installed from `nginx` user (rootless installation), run: ```shell docker pull private-registry.nginx.com/nginx-plus/rootless-agentv3: ``` For NGINX modules, run: ```shell docker pull private-registry.nginx.com/nginx-plus/modules: ``` **Note:** To protect your system and data, follow these security practices: 1. **JWTs:** Treat JSON Web Tokens (JWTs) as sensitive data. Store them securely and delete them after use to prevent unauthorized access. 2. **Shell history:** Commands with JWTs or passwords are saved in plain text in your shell history. After running such commands, clear the history to protect credentials. For example: - Edit your shell history file (such as ~/.bash_history or ~/.zsh_history) to remove specific commands. - Use `history -c` to clear all shell history in bash or zsh. ### Push the image to your private registry After pulling the image, tag it and upload it to your private registry. > **Note:** Never upload your NGINX Plus images to a public repository such as Docker Hub. Doing so violates your license agreement. Log in to your private registry: ```shell docker login ``` Tag and push the image. Replace `` with your registry’s path and `` with the your NGINX Plus version and/or OS version: ```shell docker tag private-registry.nginx.com/nginx-plus/base: /nginx-plus/base: ``` ```shell docker push /nginx-plus/base: ``` ### Run the NGINX Plus container **Note:** Starting from [NGINX Plus Release 33](nginx/releases.md#r33), the JWT file is required for each NGINX Plus instance. For more information, see [About Subscription Licenses](/solutions/about-subscription-licenses.md). To start the Docker container with NGINX Plus, you will need to pass your JWT license file named `license.jwt` as the `NGINX_LICENSE_JWT` environment variable. If the license file needs to be located in a non-default directory, specify its full path using the `NGINX_LICENSE_PATH` variable (default path: `/etc/nginx/license.jwt`). To start the Docker container with NGINX Plus only: ```sh sudo docker run \ --env=NGINX_LICENSE_JWT=$(cat license.jwt) \ --restart=always \ --runtime=runc \ -d /nginx-plus/base: ``` To start the Docker container with NGINX Plus and NGINX Agent, you will need to additionally pass the NGINX One data plane key as the `NGINX_AGENT_SERVER_TOKEN` environment variable. For more information, see [Create and manage data plane keys](/nginx-one-console/connect-instances/create-manage-data-plane-keys.md): ```sh sudo docker run \ --env=NGINX_LICENSE_JWT=$(cat license.jwt) \ --env=NGINX_AGENT_SERVER_GRPCPORT=443 \ --env=NGINX_AGENT_SERVER_HOST=agent.connect.nginx.com \ --env=NGINX_AGENT_SERVER_TOKEN="YOUR_NGINX_ONE_DATA_PLANE_KEY" \ --env=NGINX_AGENT_TLS_ENABLE=true \ --restart=always \ --runtime=runc \ -d /nginx-plus/agent: ``` where: - `NGINX_LICENSE_JWT` is your JWT license file from MyF5. The file name should be `license.jwt`. - `NGINX_AGENT_SERVER_GRPCPORT` sets a GRPC port used by NGINX Agent to communicate with NGINX Instance Manager. - `NGINX_AGENT_SERVER_HOST` sets the domain name or IP address of NGINX Instance Manager. Note that for production environments it is not recommended to expose NGINX Instance Manager to public networks. - `NGINX_AGENT_SERVER_TOKEN` sets NGINX One data plane key. See [Create and manage data plane keys](/nginx-one-console/connect-instances/create-manage-data-plane-keys.md) for details. - `NGINX_AGENT_TLS_ENABLE` enables mutual TLS, server-side TLS, or insecure mode (not recommended for production environments). See [Encrypt communication](agent/configuration/encrypt-communication.md) for details. - `YOUR_REGISTRY` is the path to your private registry. - `VERSION_TAG` is the tag assigned when pushing to your registry.
**Note:** To start the Docker Container with NGINX Plus and NGINX Agent on Debian or Ubuntu: ```sh sudo docker run \ --env=NGINX_LICENSE_JWT=$(cat license.jwt) \ --env=NGINX_AGENT_SERVER_GRPCPORT=443 \ --env=NGINX_AGENT_SERVER_HOST=agent.connect.nginx.com \ --env=NGINX_AGENT_SERVER_TOKEN="YOUR_NGINX_ONE_DATA_PLANE_KEY" \ --env=NGINX_AGENT_TLS_ENABLE=true \ --restart=always \ --runtime=runc \ -d private-registry.nginx.com/nginx-plus/agent:debian ## Using NGINX Open Source Docker Images You can create an NGINX instance in a Docker container using the NGINX Open Source image from the Docker Hub. 1. Launch an instance of NGINX running in a container and using the default NGINX configuration with the following command: ```shell docker run --name mynginx1 -p 80:80 -d nginx ``` where: - `mynginx1` is the name of the created container based on the NGINX image - the `-d` option specifies that the container runs in detached mode: the container continues to run until stopped but does not respond to commands run on the command line. - the `-p` option tells Docker to map the ports exposed in the container by the NGINX image (port `80`) to the specified port on the Docker host. The first parameter specifies the port in the Docker host, the second parameter is mapped to the port exposed in the container The command returns the long form of the container ID: `fcd1fb01b14557c7c9d991238f2558ae2704d129cf9fb97bb4fadf673a58580d`. This form of ID is used in the name of log files. 2. Verify that the container was created and is running with the `docker ps` command: ```shell $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS ... fcd1fb01b145 nginx:latest "nginx -g 'daemon of 16 seconds ago Up 15 seconds ... ... PORTS NAMES ... 0.0.0.0:80->80/tcp mynginx1 ``` This command also allows viewing the port mappings set in the previous step: the `PORTS` field in the output reports that port `80` on the Docker host is mapped to port `80` in the container. ## Creating custom NGINX Plus Docker Image As NGINX Plus is a commercial offering, NGINX Plus Docker images are not available at Docker Hub, so first you will need to create an NGINX Plus Docker image. > **Note:** Never upload your NGINX Plus images to a public repository such as Docker Hub. Doing so violates your license agreement. To generate a custom NGINX Plus image: 1. Create the Docker build context, or a Dockerfile, for example: 2. As with NGINX Open Source, default NGINX Plus image has the same default settings: - access and error logs are linked to the Docker log collector - no volumes are specified: a Dockerfile can be used to create base images from which you can create new images with volumes specified, or volumes can be specified manually: ```dockerfile VOLUME /usr/share/nginx/html VOLUME /etc/nginx ``` - no files are copied from the Docker host as a container is created: you can add `COPY` definitions to each Dockerfile, or the image you create can be used as the basis for another image 3. Log in to [MyF5 Customer Portal](https://account.f5.com/myf5). As noted in the [Prerequisites](#prerequisites], download your *nginx-repo.crt*, *nginx-repo.key*, and **JSON Web Token** files. For a trial of NGINX Plus, the files are provided with your trial package. 4. Copy the files to the directory where the Dockerfile is located. 5. Create a Docker image, for example, `nginxplus` (note the final period in the command). ```shell docker build --no-cache --secret id=nginx-key,src=nginx-repo.key --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-jwt,src=license.jwt -t nginxplus . ``` The `--no-cache` option tells Docker to build the image from scratch and ensures the installation of the latest version of NGINX Plus. If the Dockerfile was previously used to build an image without the `--no-cache` option, the new image uses the version of NGINX Plus from the previously built image from the Docker cache. 6. Verify that the `nginxplus` image was created successfully with the `docker images` command: ```shell $ docker images nginxplus REPOSITORY TAG IMAGE ID CREATED SIZE nginxplus latest ef2bf65931cf 6 seconds ago 91.2 MB ``` 7. Create a container based on this image, for example, `mynginxplus` container: ```shell docker run --name mynginxplus -p 80:80 -d nginxplus ``` 8. Verify that the `mynginxplus` container is up and running with the `docker ps` command: ```shell $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS ... eb7be9f439db nginxplus:latest "nginx -g 'daemon of 1 minute ago Up 15 seconds ... ... PORTS NAMES ... 0.0.0.0:80->80/tcp mynginxplus ``` NGINX Plus containers are controlled and managed in the same way as NGINX Open Source containers. ## Managing Content and Configuration Files Content served by NGINX and NGINX configuration files can be managed in several ways: - files are maintained on the Docker host - files are copied from the Docker host to a container - files are maintained in the container ### Maintaining Content and Configuration Files on the Docker Host When the container is created, you can mount a local directory on the Docker host to a directory in the container. The NGINX image uses the default NGINX configuration, which uses `/usr/share/nginx/html` as the container’s root directory and puts configuration files in `/etc/nginx`. For a Docker host with content in the local directory `/var/www` and configuration files in `/var/nginx/conf`, run the command: ```shell docker run --name mynginx2 \ --mount type=bind,source=/var/www,target=/usr/share/nginx/html,readonly \ --mount type=bind,source=/var/nginx/conf,target=/etc/nginx/conf,readonly \ -p 80:80 \ -d nginxplus ``` Any change made to the files in the local directories `/var/www and /var/nginx/conf` on the Docker host are reflected in the directories `/usr/share/nginx/html` and `/etc/nginx` in the container. The `readonly` option means these directories can be changed only on the Docker host, not from within the container. ### Copy content and configuration files from the Docker host Docker can copy the content and configuration files from a local directory on the Docker host during container creation. Once a container is created, the files are maintained by creating a new container when files change or by modifying the files in the container. A simple way to copy the files is to create a Dockerfile with commands that are run during generation of a new Docker image based on the NGINX image. For the file‑copy (COPY) commands in the Dockerfile, the local directory path is relative to the build context where the Dockerfile is located. Let's assume that the content directory is `content` and the directory for configuration files is `conf`, both subdirectories of the directory where the Dockerfile is located. The NGINX image has the default NGINX configuration files, including `default.conf`, in the `/etc/nginx/conf.d` directory. To use the configuration files from the Docker host only, delete the default files with the `RUN` command: ```dockerfile FROM nginx RUN rm /etc/nginx/conf.d/default.conf COPY content /usr/share/nginx/html COPY conf /etc/nginx ``` Create NGINX image by running the command from the directory where the Dockerfile is located. The period (“.”) at the end of the command defines the current directory as the build context, which contains the Dockerfile and the directories to be copied: ```shell docker build -t mynginx_image1 . ``` Create a container `mynginx3` based on the `mynginx_image1` image: ```shell docker run --name mynginx3 -p 80:80 -d mynginx_image1 ``` To make changes to the files in the container, use a helper container as described in the next section. ### Maintain content and configuration files in the container As SSH cannot be used to access the NGINX container, to edit the content or configuration files directly you need to create a helper container that has shell access. For the helper container to have access to the files, create a new image that has the proper Docker data volumes defined for the image: 1. Copy nginx content and configuration files and define the volume for the image with the Dockerfile: ```dockerfile FROM nginx COPY content /usr/share/nginx/html COPY conf /etc/nginx VOLUME /usr/share/nginx/html VOLUME /etc/nginx ``` 2. Create the new NGINX image by running the following command: ```shell docker build -t mynginx_image2 . ``` 3. Create an NGINX container `mynginx4` based on the `mynginx_image2` image: ```shell docker run --name mynginx4 -p 80:80 -d mynginx_image2 ``` 4. Start a helper container `mynginx4_files` that has a shell, providing access the content and configuration directories of the `mynginx4` container we just created: ```shell $ docker run -i -t --volumes-from mynginx4 --name mynginx4_files debian /bin/bash root@b1cbbad63dd1:/# ``` where: - the new `mynginx4_files` helper container runs in the foreground with a persistent standard input (the `-i` option) and a tty (the `-t` option). All volumes defined in `mynginx4` are mounted as local directories in the helper container. - the `debian` argument means that the helper container uses the Debian image from Docker Hub. Because the NGINX image also uses Debian, it is most efficient to use Debian for the helper container, rather than having Docker load another operating system - the `/bin/bash` argument means that the bash shell runs in the helper container, presenting a shell prompt that you can use to modify files as needed To start and stop the container, run the commands: ```shell docker start mynginx4_files docker stop mynginx4_files ``` To exit the shell but leave the container running, press `Ctrl+p` followed by `Ctrl+q`. To regain shell access to a running container, run this command: ```shell docker attach mynginx4_files ``` To exit the shell and terminate the container, run the `exit` command. ## Manage logging You can use default logging or customize logging. ### Use default logging By default, the NGINX image is configured to send NGINX [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) and [error log](https://nginx.org/en/docs/ngx_core_module.html#error_log) to the Docker log collector. This is done by linking them to `stdout` and `stderr`: all messages from both logs are then written to the file `/var/lib/docker/containers/container-ID/container-ID-json.log` on the Docker host. The container‑ID is the long‑form ID returned when you [create a container](#docker_oss_image). To display the long form ID, run the command: ```shell docker inspect --format '{{ .Id }}' container-name ``` You can use both the Docker command line and the Docker Engine API to extract the log messages. To extract log messages from the command line, run the command: ```shell docker logs container-name ``` To extract log messages using the Docker Remote API, send a `GET` request using the Docker Unix sock: ```shell curl --unix-sock /var/run/docker-sock http://localhost/containers/container-name/logs?stdout=1&stderr=1 ``` To include only access log messages in the output, include only `stdout=1`. To limit the output to error log messages, include only `stderr=1`. For other available options, see [Get container logs](https://docs.docker.com/reference/api/engine/version/v1.45/#tag/Container/operation/ContainerLogs) section of the [Docker Engine API](https://docs.docker.com/reference/api/engine/version/v1.45/) documentation. ### Use customized logging If you want to configure logging differently for certain configuration blocks (such as `server {}` and `location {}`), define a Docker volume for the directory in which to store the log files in the container, create a helper container to access the log files, and use any logging tools. To implement this, create a new image that contains the volume or volumes for the logging files. For example, to configure NGINX to store log files in `/var/log/nginx/log`, add a `VOLUME` definition for this directory to the Dockerfile (provided that content and configuration Files are [managed in the container](#manage_container)): ```dockerfile FROM nginx COPY content /usr/share/nginx/html COPY conf /etc/nginx VOLUME /var/log/nginx/log ``` Then you can [create an image](#docker_plus_image) and use it to create an NGINX container and a helper container that have access to the logging directory. The helper container can have any desired logging tools installed. ## Control NGINX Since there is no direct access to the command line of the NGINX container, NGINX commands cannot be sent to a container directly. Instead, [signals](https://nginx.org/en/docs/control.html) can be sent to a container via Docker `kill` command. To reload the NGINX configuration, send the `HUP` signal to Docker: ```shell docker kill -s HUP container-name ``` To restart NGINX, run this command to restart the container: ```shell docker restart container-name ``` -------------------------------------------------------------------------------- title: "gRPC Health Checks" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/grpc-health-check/index.md description: Monitor the health of gRPC servers in an upstream group by sending periodic health checks. -------------------------------------------------------------------------------- ## Introduction F5 NGINX Plus can monitor the health of upstream servers by making active health checks. NGINX Plus R23 supports the [gRPC health checking protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md#grpc-health-checking-protocol) so that upstream gRPC services can be tested for their ability to handle new requests. This is particularly important in dynamic and containerized environments. When adding a new instance of a gRPC service, it is important that requests are sent to the fully operating service. ## Prerequisites - You have configured an upstream group of servers that handles gRPC network traffic and specified a shared memory zone that keeps the state of these servers, for example: ```nginx stream { #... upstream grpc_backend { zone grpc_backend 64k; server 10.0.0.1:50051; server 10.0.0.2:50051; } #... } ``` - You have configured a server that routes gRPC requests to the upstream server group: ```nginx location /grpc { grpc_pass grpc://grpc_backend; } ``` ## gRPC Servers that accept health checking protocol If your gRPC services support the gRPC health checking protocol, specify the [`type=grpc`](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_grpc) parameter of the `health_check` directive: ```nginx location / { grpc_pass grpc://grpc_backend; health_check mandatory type=grpc; } ``` In this example, according to health checking protocol, the `Check` method of the `Health` service will be invoked, and the gRPC server that respond with `SERVING` are considered healthy. The `mandatory` parameter ensures that the health check must pass before traffic is sent on an instance, for example, when it is introduced or reloaded. Note that the `type=grpc` must be specified after all other `health_check` parameters. If there are several gRPC services exposed on each upstream server then the most significant service can be monitored by specifying the service name with the `grpc_service` parameter: ```nginx location / { grpc_pass grpc://grpc_backend; health_check mandatory type=grpc grpc_service=MyStatus; } ``` ## gRPC Servers that do not accept health checking protocol If your gRPC services do not implement the gRPC health checking protocol, it is still possible to perform health check by sending the `Check` method and expecting a particular [`status code`](https://github.com/grpc/grpc/blob/master/doc/statuscodes.md#status-codes-and-their-use-in-grpc). This will test that the upstream server is responding to gRPC requests. Specify the gRPC error response code with the `grpc_status` parameter: ```nginx location / { grpc_pass grpc://grpc_backend; health_check type=grpc grpc_status=12; # 12=unimplemented } ``` In this case the response with the gRPC status of `12` / `UNIMPLEMENTED` will be considered healthy. Note that the `type=grpc` parameter is not compatible with `uri` or `match` parameters of the `health_check` directive. The `type=grpc` parameter must be specified after all other directive parameters: `grpc_service` and `grpc_status` must follow `type=grpc`. ## See also - [Deploying NGINX Plus as an API Gateway, Part 3: Publishing gRPC Services](https://www.nginx.com/blog/deploying-nginx-plus-as-an-api-gateway-part-3-publishing-grpc-services/) -------------------------------------------------------------------------------- title: "HTTP CONNECT forward proxy" url: https://docs.nginx.com/nginx/admin-guide/web-server/http-connect-proxy/index.md description: HTTP traffic tunneling via the HTTP CONNECT method, enabling centralized egress control through a trusted NGINX Plus server. -------------------------------------------------------------------------------- In corporate networks, NGINX Plus R36 and later can be configured as a forward proxy server, facilitating client access to external resources. A forward proxy operates between internal clients and the global network, enabling centralized traffic control. Unlike a reverse proxy that protects servers, upstreams, or services, forward proxy serves client requests and regulates their access to the external resources. To enable this functionality, the HTTP `CONNECT` method is used to establish a secure tunnel between the client and the proxy server (NGINX Plus). This tunnel permits the transmission of HTTPS traffic and other protocols, such as SSH or FTP, through the proxy. ## Enable HTTP CONNECT proxy To enable the HTTP `CONNECT` forward proxy, add the [`tunnel_pass`](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html#tunnel_pass) directive that turns on the forward proxy functionality within the `server` or `location` blocks. The `tunnel_pass` directive can be used without any parameters. The default value is [`$host`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_host):[`$request_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_port) variables: all requests are automatically forwarded to external resources based on the host and the port which we are trying to access. The `tunnel_pass` directive is a content handling directive (similar to `proxy_pass`), and if specified for the `server` block, will not be inherited by corresponding `location` blocks. If you have a location that implements any other NGINX Plus capabilities within the forward proxy, you will need to enable the `tunnel_pass` for this `location` as well. See [Disable other HTTP methods](#disable-other-http-methods). ```nginx server { listen 10.10.1.11:3128; tunnel_pass; } ``` In this basic configuration, the client establishes a tunnel with NGINX Plus, sends the HTTP `CONNECT` method to the specified port `3128` which is standard for forward proxying, and if successful, responds with `200 OK`. The tunnel is established and the client can perform the TLS handshake and get the response. ## Disable other HTTP methods As soon as the `CONNECT` method is enabled, all other methods such as `GET`, `POST` are not applicable in this mode. Rejecting them ensures that only tunnel requests are allowed thus improving security. You can create a rule with the [`if`](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if) directive to deny all HTTP methods except `CONNECT`, If a client uses any other method, NGINX Plus returns the `403` error code: ```nginx server { listen 10.10.1.11:3128; # Handle other methods location / { if ($request_method != CONNECT) { return 403 "Forbidden: allows only CONNECT method"; } # allow CONNECT requests tunnel_pass; } } ``` Note that in the example the [`tunnel_pass`](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html#tunnel_pass) directive is specified inside the `location` block. Since `tunnel_pass` is a content handling directive, when it is specified for the `server` block, it is not inherited by nested locations. ## Enable logging For testing, debugging and monitoring purposes you can configure error and access logs. 1. On the `http` level, specify the [`log_format`](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) directive, give the log a name (for example, `http_connect`), and include the metrics: ```nginx log_format http_connect '$remote_addr [$time_local] ' '$request_method $host:$request_port $status ' '$upstream_addr $bytes_sent $upstream_connect_time ' '$request_time "$http_user_agent"'; ``` See the [NGINX variables index](https://nginx.org/en/docs/varindex.html) for the list of all supported variables. 2. Specify the [`error_log`](https://nginx.org/en/docs/ngx_core_module.html#error_log) and [`access_log`](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directives on the same configuration level where the [`tunnel_pass`](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html#tunnel_pass) directive is defined: ```nginx access_log logs/connect_access.log http_connect; error_log logs/connect_error.log; ``` ## Test the configuration This example sums up the steps above: it enables logging for the `location` block which denies all HTTP methods except `CONNECT` and where the `tunnel_pass` directive is specified: ```nginx log_format http_connect '$remote_addr [$time_local] ' '$request_method $host:$request_port $status ' '$upstream_addr $bytes_sent $upstream_connect_time ' '$request_time "$http_user_agent"'; server { listen 10.10.1.11:3128; location / { if ($request_method != CONNECT) { return 403 "Forbidden: allows only CONNECT method"; } tunnel_pass; access_log logs/connect_access.log http_connect; error_log logs/connect_error.log; } } ``` 1. Test NGINX configuration for syntax validity and apply the changes: ```nginx nginx -t && nginx -s reload ``` 2. Send a test request with the `curl` command: ```shell curl -v -x 10.10.1.11:3128 https://example.com ``` 3. Check the access log to see the result of the command: ```shell tail -n 100 /logs/connect_access.log ``` The result of the command: ```none logs > connect_access.log 1 10.10.1.240 [01/Dec/2025:10:46:27 +0000] CONNECT example.com:443 200 127.456.789.0:443 4275 0.014 0.258 "curl/8.7.1" ``` The result shows that the client (`10.10.1.240`) successfully established a connection (`200`) to the destination server (`example.com:443` resolved to `127.456.789.0:443`). ## Access control It is highly recommended to restrict access to the proxy servers. Access control can be managed in several ways: - by ports and port ranges with the [`num_map`](https://nginx.org/en/docs/http/ngx_http_num_map_module.html) module (NGINX Plus R36) - by IP addresses with the [`geo`](https://nginx.org/en/docs/http/ngx_http_geo_module.html) module - by hostnames with the [`map`](https://nginx.org/en/docs/http/ngx_http_map_module.html) module All of these access methods can be used together and combined with the the [`proxy_allow_upstream`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_allow_upstream) directive (NGINX Plus R36). ### Restrict by ports and port ranges The [`num_map`](https://nginx.org/en/docs/http/ngx_http_num_map_module.html) module introduced in NGINX Plus R36 allows defining ports and port ranges and uses the same approach as the `map` block. For example, if the value of the [`$request_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_port) variable is `443`, the `$allowed_port` variable is assigned the value `ssl`. It also restricts the allowed port range for the upstream connections or proxy requests: ```nginx num_map $request_port $allowed_port { default 0; 443 ssl; <=1023 less-eq; 8080-8090 range; >8092 more; } ``` ### Restrict by IP addresses Access by IP addresses can be restricted with the [`geo`](https://nginx.org/en/docs/http/ngx_http_geo_module.html) module: ```nginx geo $allowed_networks { 10.10.1.0/24 allow; 10.20.11.0/24 allow; default deny; } ``` ### Restrict by hostnames Restriction by hostnames can be configured with the [`map`](https://nginx.org/en/docs/http/ngx_http_map_module.html) module, which works mainly with strings and regular expressions. Although it is possible to match port numbers using `map`, it is recommended to use the [`num_map`](https://nginx.org/en/docs/http/ngx_http_num_map_module.html) module for more port-based restrictions, as it provides additional functionality such as defining ranges. In this example, access is allowed to `example.com` and its subdomains only: ```nginx map $host $allowed_host { hostnames; default 0; example.com 1; *.example.com 1; } ``` ### Combine access control methods These access methods can be combined in the [`tunnel_allow_upstream`](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html#tunnel_allow_upstream) directive, which performs access checks when the upstream server to which the request will be sent is selected. If any variable passed to [`tunnel_allow_upstream`](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html#tunnel_allow_upstream) evaluates to non-zero or non-empty value, it will be calculated as `1` and access is granted. If all variables evaluate to `0` or an empty value, access is denied. ### Access control example ```nginx log_format connect_debug '$remote_addr [$time_local] ' '$request_method $host:$request_port $status ' '$upstream_addr $bytes_sent $upstream_connect_time ' '$request_time "$http_user_agent"'; num_map $request_port $allowed_port { default 0; 443 ssl; <=1023 less-eq; 8080-8090 range; >8092 more; } geo $allowed_networks { 10.10.1.0/24 allow; 10.20.11.0/24 allow; default deny; } map $host $allowed_host { hostnames; default 0; example.com 1; *.example.com 1; } server { listen 10.10.1.11:3128; #Allow CONNECT only to certain ports/nets/hosts tunnel_allow_upstream $allowed_networks $allowed_port $allowed_host; error_page 403; satisfy all; allow 10.10.0.0/16; allow 127.0.0.1; deny all; tunnel_pass; location ^~ /errors/ { internal; root html; allow all; } access_log logs/connect_access.log connect_debug; error_log logs/connect_debug.log debug; } ``` Access can also be limited using other modules, for example with the [`satisfy`](https://nginx.org/en/docs/http/ngx_http_core_module.html#satisfy) directive, or the [`allow`/`deny`](https://nginx.org/en/docs/http/ngx_http_access_module.html#allow) directives. ## mTLS authentication Mutual TLS (mTLS) ensures that both the client and server authenticate each other by exchanging and validating trusted certificates. Once the authentication succeeds, a secure, encrypted connection is established. For an HTTP CONNECT proxy, mutual TLS (mTLS) is currently the only supported authentication method. Other methods, such as [basic authentication](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) are currently not supported because NGINX Plus processes only the `Authorization` header, not the `Proxy-Authorization` header. To enable mTLS, you need: - a server certificate and private key, used by NGINX Plus to authenticate itself to clients - a client Certificate Authority (CA) certificate, used by NGINX Plus to verify client certificates - one or more client certificates, signed by the trusted client CA, used by clients to authenticate to the proxy - NGINX Plus configuration that enables TLS and enforces client certificates verification Certificates can be created using tools such as OpenSSL or obtained from certificate providers such as Let's Encrypt. ### Configure mTLS In your NGINX Plus configuration file: 1. Enable the TLS layer on proxy listener by adding the `ssl` parameter to the `listen` directive, port `3128` is a commonly used port for forward proxies: ```nginx listen 10.10.1.11:3128 ssl; #... ``` 2. Specify the proxy server SSL certificate and private key (the certificate that NGINX Plus presents to clients): ```nginx #... ssl_certificate /etc/ssl/certs/forward_proxy_server.crt; ssl_certificate_key /etc/ssl/certs/forward_proxy_server.key; #... ``` 3. Configure client certificate verification. Specify the trusted Client CA certificate used to validate client certificates with the [`ssl_client_certificate`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_client_certificate) directive. With [`ssl_verify_client`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client) set to on, every client is required to present a valid certificate during the TLS handshake, otherwise the connection will be rejected: ```nginx #... ssl_client_certificate /etc/ssl/certs/forward_proxy_client_ca.crt; ssl_verify_client on; ssl_verify_depth 1; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; #... ``` ### Conditional forward proxy access based on mTLS Based on the result of client certificate verification, you can implement different access policies. This scenario enables conditional access: if a client provides a valid certificate, access is granted to all resources through the proxy. If not, access is limited to particular resources or blocked. 1. Set the [`ssl_verify_client`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client) directive value to `optional` to allow clients to connect with or without a client certificate, making conditional access possible: ```nginx #... ssl_client_certificate /etc/ssl/certs/forward_proxy_client_ca.crt; ssl_verify_client optional; #... ``` 2. Verify the status of the client certificate using the [`map`](https://nginx.org/en/docs/http/ngx_http_map_module.html#map) block and the [`$ssl_client_verify`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_verify) variable. The result will be saved to the `$ssl_ok` variable: ```nginx map $ssl_client_verify $ssl_ok { default 0; SUCCESS 1; } ``` In the `map` block, the value of the `$ssl_ok` variable depends on the status of [`$ssl_client_verify`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_verify) variable. If the result of client certificate verification is “SUCCESS”, the `$ssl_ok` variable gets the value `1`. 3. Check the hostname the client is going to access based on the `Host` header of the request (the [$host](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_host) variable). Access will be allowed only for `example.com` and its subdomains. The `hostnames` parameter of the [`map`](https://nginx.org/en/docs/http/ngx_http_map_module.html#map) directive is required to recognize hostnames and wildcards in the hostnames: ```nginx map $host $allowed_host { hostnames; default 0; example.com 1; *.example.com 1; } ``` 4. Define access rules by combining the results of client certificate verification (`$ssl_ok`) and hostname validation (`$allowed_host`). To allow access to all websites, the client must present a valid client certificate. If no client certificate is provided, access is restricted to `example.com` and its subdomains. The result is stored in the `$connect_ok` variable: ```nginx map "$ssl_ok:$allowed_host" $connect_ok { default 0; "1:0" 1; "1:1" 1; "0:1" 1; } ``` 5. Restrict upstream connections to valid `CONNECT` requests using the [`tunnel_allow_upstream`](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html#tunnel_allow_upstream) directive based on the result in the `$connect_ok` variable: ```nginx tunnel_allow_upstream $connect_ok; tunnel_pass; ``` ### Complete example This configuration sums up the steps and enables authorization based on client certificate verification result. Clients with valid client certificates are granted full access through the proxy. Clients without a certificate are restricted to `example.com` and its subdomains, all other `CONNECT` requests are denied. ```nginx log_format http_connect '$remote_addr [$time_local] ' '$request_method $host:$request_port $status ' '$upstream_addr $bytes_sent $upstream_connect_time ' '$request_time "$http_user_agent"'; map $ssl_client_verify $ssl_ok { default 0; SUCCESS 1; } map $host $allowed_host { hostnames; default 0; example.com 1; *.example.com 1; } map "$ssl_ok:$allowed_host" $connect_ok { default 0; "1:0" 1; "1:1" 1; "0:1" 1; } server { listen 10.10.1.11:3128 ssl; ssl_certificate /etc/ssl/certs/forward_proxy_server.crt; ssl_certificate_key /etc/ssl/certs/forward_proxy_server.key; ssl_client_certificate /etc/ssl/certs/forward_proxy_client_ca.crt; ssl_verify_client optional; ssl_verify_depth 1; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; tunnel_allow_upstream $connect_ok; tunnel_pass; access_log logs/connect_access.log http_connect; error_log logs/connect_error.log; } ``` The first `map` block checks if the client passed the mTLS verification. It sets the `$ssl_ok` variable to `1` if NGINX Plus successfully verified the client certificate during the TLS handshake, and to `0` otherwise. The second `map` block checks if the requested host is allowlisted: if the `CONNECT` target hostname is `example.com` or any of its subdomains, the `$allowed_host` variable is set to `1` and to `0` otherwise. The third `map` block builds the final decision variable, `$connect_ok`. Its result will be evaluated by the `tunnel_allow_upstream` directive: if it has a non-empty value, access will be granted, with restrictions above implemented. If the client cert is valid (`$ssl_ok`=`1`), access is allowed regardless of host. If the client cert is not valid (`$ssl_ok`=`0`), access is allowed only if the host is allowlisted. Otherwise access is denied (default is `0`). The [`ssl_verify_client`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client) directive is set to `optional` which allows clients to connect with or without client certificate, making conditional access possible. #### Testing 1. Test the nginx configuration for syntax validity and apply the configuration: ```nginx nginx -t && \ nginx -s reload ``` 2. Test the proxy authentication layer for requests to verify that the proxy is protected as expected under different scenarios. - Client certificate is not provided: ```shell curl -v -x https://myproxy:3128 \ https://www.example.com \ --proxy-cacert ./conf/tls/ca.crt ``` where: - `myproxy:3128` is the proxy address and port - `https://www.example.com` is the destination URL requested through the proxy - `--proxy-cacert ./conf/tls/ca.crt` - the CA used to verify the proxy’s server certificate. Expected result: requests to `example.com` and its subdomains succeed (for example, `301 Moved Permanently`), requests to other domains are denied (for example, `502 Bad Gateway`). - Correct client certificate is provided (the `--proxy-cert` and `--proxy-key` parameters): ```shell curl -v -x https://myproxy:3128 \ https://www.example.com \ --proxy-cert ./conf/tls/client.crt \ --proxy-key ./conf/tls/client.key \ --proxy-cacert ./conf/tls/ca.crt ``` For requests to all resources, the request should succeed (for example, `301 Moved Permanently`). ## See also - [Secure and harden forward proxies in NGINX Plus](https://community.f5.com/kb/technicalarticles/secure-and-harden-forward-proxies-in-nginx-plus/344989) - [F5 NGINX Plus R36 release blogpost](https://community.f5.com/kb/technicalarticles/f5-nginx-plus-r36-release-now-available/344514) - [`ngx_http_tunnel_module` module reference](https://nginx.org/en/docs/http/ngx_http_tunnel_module.html) -------------------------------------------------------------------------------- title: "Limiting Access to Proxied HTTP Resources" url: https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/index.md description: Protect your upstream web and application servers by limiting connections, rate of requests, or bandwidth, based on client IP address or other variables. -------------------------------------------------------------------------------- This article explains how to set the maximum number of requests for a connection, or the maximum rate of downloading content from the server. ## Introduction Using NGINX and F5 NGINX Plus, it is possible to limit: - The number of connections per key value (for example, per IP address) - The request rate per key value (the number of requests that are allowed to be processed during a second or minute) - The download speed for a connection Note that IP addresses can be shared behind NAT devices, so limiting by IP address should be used judiciously. ## Limiting the Number of Connections To limit the number of connections: 1. Use the [limit_conn_zone](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn_zone) directive to define the key and set the parameters of the shared memory zone (the worker processes will use this zone to share counters for key values). As the first parameter, specify the expression evaluated as a key. In the second parameter `zone`, specify the name of the zone and its size: ```nginx limit_conn_zone $binary_remote_addr zone=addr:10m; ``` 2. Use the [limit_conn](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn) directive to apply the limit within the `location {}`, `server {}`, or `http {}` context. Specify the name of the shared memory zone as the first parameter, and the number of allowed connection per key as the second parameter: ```nginx location /download/ { limit_conn addr 1; } ``` The number of connections is limited on an IP address basis because the [`$binary_remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_binary_remote_addr) variable is used as a key. Another way to limit the number of connections for a given server is by using the [`$server_name`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_server_name) variable: ```nginx http { limit_conn_zone $server_name zone=servers:10m; server { limit_conn servers 1000; } } ``` ## Limiting the Request Rate Rate limiting can be used to prevent DDoS attacks, or prevent upstream servers from being overwhelmed by too many requests at the same time. The method is based on the [`leaky bucket`](https://en.wikipedia.org/wiki/Leaky_bucket) algorithm: requests arrive at the bucket at various rates and leave the bucket at fixed rate. Before using rate limiting, you will need to configure global parameters of the "leaky bucket": - key - a parameter used to differentiate one client from another, generally a variable - shared memory zone - the name and size of the zone that keeps states of these keys (the "leaky bucket") - rate - the request rate limit specified in requests per second (`r/s`) or requests per minute (`r/m`) ("leaky bucket draining"). Requests per minute are used to specify a rate less than one request per second. These parameters are set with the [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive. The directive is defined on the `http {}` level - such approach allows applying different zones and request overflow parameters to different contexts: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; } ``` With this configuration, the shared memory zone `one` with the size of 10 megabytes is created. The zone keeps states of client IP addresses set with the [`$binary_remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_binary_remote_addr) variable. Note that in comparison to [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr) which also holds a client’s IP address, [`$binary_remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_binary_remote_addr) holds the binary representation of IP address which is shorter. The optimal size of the shared memory zone can be counted using the following data: the size of [`$binary_remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_binary_remote_addr) value is 4 bytes for IPv4 addresses, stored state occupies 128 bytes on 64-bit platforms. Thus, state information for about 16,000 IP addresses occupies 1 megabyte of the zone. If storage is exhausted when NGINX needs to add a new entry, it removes the oldest entry. If the space freed is still not enough to accommodate the new record, NGINX returns status code `503 Service Unavailable`. The status code can be redefined with the [limit_req_status](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_status) directive. Once the zone is set, you can use requests limiting anywhere in the NGINX configuration with the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) specified for a `server {}`, `location {}`, or `http {}` context: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; server { #... location /search/ { limit_req zone=one; } } } ``` With this configuration, NGINX will process no more than `1` request per second within the `/search/` location. Processing of these requests is delayed in such a way that the overall rate is not greater than specified. If the number of requests exceeds the specified rate, NGINX will delay processing of such requests until the "bucket" (shared memory zone `one`) is full. For requests that arrive at the full bucket, NGINX will respond with the `503 Service Unavailable` error (if not redefined with [limit_req_status](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_status)). ### Testing the Request Rate Limit Before configuring real-life rate limiting, you can try the “dry run” mode that does not limit the requests processing rate. However, such excessive requests are still accounted in the shared memory zone and logged. The “dry run” mode can be enabled with the [limit_req_dry_run](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_dry_run) directive: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; server { #... location /search/ { limit_req zone=one; limit_req_dry_run on; } } } ``` Every request that would exceed the defined rate limit will be logged with the “dry run” mark: ```none 2019/09/03 10:28:45 [error] 142#142: *13246 limiting requests, dry run, excess: 1.000 by zone "one", client: 172.19.0.1, server: www.example.com, request: "GET / HTTP/1.0", host: "www.example.com:80" ``` ### Handling Excessive Requests Requests are limited to fit the rate defined in the [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive. If the number of requests exceeds the specified rate and the shared memory zone becomes full, NGINX will respond with an error. As traffic tends to be bursty, returning the error in response to a client request during traffic bursts is not the best case. Such excessive requests in NGINX can be buffered and processed. The `burst` parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive sets the maximum number of excessive requests that await to be processed at the specified rate: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; server { #... location /search/ { limit_req zone=one burst=5; } } } ``` With this configuration, if request rate exceeds `1` request per second, requests beyond the rate will be put into the zone `one`. When the zone is full, excessive requests will be queued (`burst`), the size of this queue is `5` requests. Request processing in the queue is delayed in such a way that the overall rate is not greater than specified. Requests above the burst limit will be rejected with the `503` error. If delaying of request is not desired during traffic burst, add the `nodelay` parameter: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; server { #... location /search/ { limit_req zone=one burst=5 nodelay; } } } ``` With this configuration, excessive requests within the `burst` limit will be served immediately regardless of the specified `rate`, requests above the burst limit will be rejected with the `503` error. ### Delaying Excessive Requests Another way to handle excessive requests is to serve some number of these requests without delay, then apply rate limiting up to the point when excessive requests will be rejected. This can be achieved with the `delay` and `burst` parameters. The `delay` parameter defines the point at which excessive requests are delayed to comply with the defined rate limit: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; server { #... location /search/ { limit_req zone=one burst=5 delay=3; } } } ``` With this configuration, first 3 requests (`delay`) are passed without delay, next 2 requests (`burst` - `delay`) are delayed in such a way that the overall rate is not greater than specified, further excessive requests will be rejected because the total burst size has been exceeded, subsequent requests will be delayed. ### Synchronizing Contents of Many Shared Memory Zones If you have a computer cluster with several NGINX instances and these instances use the `limit_req` method, it is possible to sync the contents of their shared memory zones on conditions that: - the [zone_sync](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync) functionality is configured for each instance - shared memory zones set in the [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive for each instance have the same name - the [sync](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone_sync) parameter of the [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive is specified for each instance: ```nginx http { #... limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s sync; } ``` See [Runtime State Sharing in a Cluster](nginx/admin-guide/high-availability/zone_sync.md) for details. ## Limiting the Bandwidth To limit the bandwidth per connection, use the [limit_rate](https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate) directive: ```nginx location /download/ { limit_rate 50k; } ``` With this setting a client will be able to download content through a single connection at a maximum speed of `50` kilobytes per second. However, the client can open several connections. So if the goal is to prevent a speed of downloading greater than the specified value, the number of connections should also be limited. For example, one connection per IP address (if the shared memory zone specified above is used): ```nginx location /download/ { limit_conn addr 1; limit_rate 50k; } ``` To impose the limit only after the client downloads a certain amount of data, use the [limit_rate_after](https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate_after) directive. It may be reasonable to allow a client to quickly download a certain amount of data (for example, a file header — film index) and limit the rate for downloading the rest of the data (to make users watch a film, not download). ```nginx limit_rate_after 500k; limit_rate 20k; ``` The following example shows the combined configuration for limiting the number of connections and the bandwidth. The maximum allowed number of connections is set to `5` connections per client address, which fits most common cases since modern browsers typically open up to 3 connections at a time. Meanwhile the location that serves downloads allows only one connection: ```nginx http { limit_conn_zone $binary_remote_addr zone=addr:10m; server { root /www/data; limit_conn addr 5; location / { } location /download/ { limit_conn addr 1; limit_rate_after 1m; limit_rate 50k; } } } ``` ### Dynamic Bandwidth Control The [limit_rate](https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate) value can also be specified as a variable - this enables dynamic bandwidth use cases, for example, allow a higher bandwidth limit to modern browsers: ```nginx map $ssl_protocol $response_rate { "TLSv1.1" 10k; "TLSv1.2" 100k; "TLSv1.3" 1000k; } server { listen 443 ssl; ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3; ssl_certificate www.example.com.crt; ssl_certificate_key www.example.com.key; location / { limit_rate $response_rate; # Limit bandwidth based on TLS version limit_rate_after 512; # Apply limit after headers have been sent proxy_pass http://my_backend; } } ``` ## See Also - [Rate Limiting with NGINX and NGINX Plus](https://www.nginx.com/blog/rate-limiting-nginx/) -------------------------------------------------------------------------------- title: "NGINX Plus FIPS Compliance" url: https://docs.nginx.com/nginx/fips-compliance-nginx-plus/index.md -------------------------------------------------------------------------------- ## What is FIPS The Federal Information Processing Standard (FIPS), issued by the [U.S. National Institute of Standards and Technology](https://www.nist.gov/) (NIST), defines mandatory security requirements for cryptographic modules used in federal IT systems. [FIPS 140-2](https://csrc.nist.gov/pubs/fips/140-2/upd2/final), and its successor [FIPS 140-3](https://csrc.nist.gov/pubs/fips/140-3/final), establish strict standards to protect sensitive information, including government communications and citizen data. ## Why FIPS-140 matters FIPS 140 is a mandatory cryptographic standard in the United States and Canada for federal agencies, their contractors, and many regulated industries. Non-compliance can result to contract loss, restricted project access, fines, or, in severe cases, data breaches compromising personal information or national security. Some industries such as finance, healthcare, energy, also adopt FIPS to enhance data protection and operational security. ### FIPS compliance in U.S. Currently, both FIPS 140-2 and FIPS 140-3 certifications are accepted. However, FIPS 140-2 is being phased out as part of the [FIPS 140-3 transition plan](https://csrc.nist.gov/projects/fips-140-3-transition-effort). After September 22, 2026, only FIPS 140-3 certifications will be recognized. Organizations are encouraged to migrate to FIPS 140-3 to meet updated cryptographic security requirements. | **Sector / Program** | **Version** | **Status** | |--------------------------------|----------------|---------------| | **Federal Programs** | | | | CJIS | 140-2 or 140-3 | Mandatory | | FedRAMP | 140-2 or 140-3 | Mandatory | | FISMA | 140-2 or 140-3 | Mandatory | | DFARS | 140-2 or 140-3 | Mandatory | | DoDIN APL | 140-2 or 140-3 | Mandatory | | FAA | 140-2 to 140-3 | Transitioning | | TSA | 140-2 or 140-3 | Recommended | | **Defense & Intelligence** | | | | CMMC | 140-2 or 140-3 | Mandatory | | Intelligence Community | 140-2 to 140-3 | Transitioning | | NSA CSfC | 140-2 to 140-3 | Transitioning | | Military & Tactical Systems | 140-2 to 140-3 | Transitioning | | **Healthcare & Education** | | | | HIPAA | 140-2 or 140-3 | Mandatory | | HITECH | 140-2 or 140-3 | Mandatory | | Department of Veterans Affairs | 140-2 or 140-3 | Mandatory | | FERPA | 140-2 or 140-3 | Recommended | | **Commercial/Private Sector** | | | | PCI DSS | 140-2 or 140-3 | Recommended | | Common Criteria | 140-2 or 140-3 | Recommended | | **Infrastructure & Critical Systems** | | | | Critical Infrastructure | 140-2 or 140-3 | Recommended | | Nuclear Regulatory Commission | 140-2 or 140-3 | Recommended | | **State & Local Government** | | | | State and Local Gov Programs | 140-2 or 140-3 | Mandatory | ### FIPS compliance in other countries Although FIPS 140 is primarily a North American government cryptographic standard, it is widely recognized as a global benchmark for cryptographic security. Numerous countries outside North America align their cryptographic requirements with FIPS, especially in regulated sectors such as finance, defense, healthcare, and critical infrastructure. | Country/Region | FIPS Use | |----------------|-----------------------------------------------------------------------------| | Australia | Referenced for government, defense, and cryptography systems. | | Canada | Mandatory for federal and sensitive systems. | | Denmark | Referenced in finance, healthcare, and NATO communications. | | Estonia | Adopted for e-governance and critical systems. | | Finland | Relied on for defense and NATO communications. | | France | Relied on for defense and secure systems. | | Germany | Relied on for defense, critical infrastructure, and NATO communications. | | Israel | Trusted in defense, government, and financial systems. | | Italy | Relied on for defense and financial cryptography. | | Japan | Referenced in government and financial cryptographic practices. | | Netherlands | Referenced in finance, healthcare, and NATO communications. | | New Zealand | Referenced for government and national cryptography. | | Poland | Relied on for secure government and NATO communications. | | Spain | Referenced in NATO communications and critical systems. | | Sweden | Relied on for defense and secure NATO communications. | | UAE | Trusted in finance, energy, and interoperability with the U.S. cryptography.| | United Kingdom | Referenced for defense, health, and procurement standards. | | United States | Mandatory for federal government systems and contractors. | ## FIPS compliant vs FIPS validated FIPS validation is a formal multistep process that certifies cryptographic modules through testing under the [Cryptographic Module Validation Program](https://csrc.nist.gov/Projects/cryptographic-module-validation-program/cmvp-flow) (CMVP). The process is managed by the [NIST](https://csrc.nist.gov/) and requires accredited third-party laboratories to evaluate the cryptographic module. Once a module passes validation, it is officially recognized as FIPS-validated (FIPS-certified). FIPS compliance indicates that a system or a module claims to meet the FIPS requirements, however, it has not been officially tested or certified under the CMVP program. ## FIPS compliance with NGINX Plus NGINX Plus is **FIPS 140-2 Level 1** and **FIPS 140-3 Level 1 compliant**, provided that the operating system and the OpenSSL library are operating in FIPS mode. NGINX Plus relies exclusively on the operating system’s FIPS-validated OpenSSL library for all SSL/TLS, HTTP/2, and HTTP/3 encryption and decryption operations. ## FIPS compliance with NGINX Open Source While NGINX Plus is tested to work on FIPS-enabled operating systems in FIPS mode, NGINX Open Source is not verified for such environments, especially when third-party builds or modules implementing custom cryptographic functions are used. Compiling NGINX Open Source for FIPS mode may also require additional OS-level dependencies beyond its core requirements, potentially introducing unintended risks. Organizations should consult their security and compliance teams to ensure their configurations meet FIPS requirements. ## FIPS validation of operating systems Several operating system vendors have obtained FIPS 140-2 Level 1 and 140-3 Level 1 validation for the OpenSSL Cryptographic Module included with their respective operating systems: - RedHat: [RedHat FIPS Certifications](https://access.redhat.com/compliance/fips) - Ubuntu: [Overview of FIPS-certified modules](https://documentation.ubuntu.com/security/docs/compliance/fips/fips-overview/) - Oracle: [Oracle FIPS Certifications](https://www.oracle.com/corporate/security-practices/assurance/development/external-security-evaluations/fips/certifications/) - SUSE: [SUSE FIPS 140-3 cryptographic certificates](https://www.suse.com/c/suse-has-received-first-fips-140-3-cryptographic-certificates/) - AWS: [FIPS 140-3 Compliance](https://aws.amazon.com/compliance/fips/) - Amazon Linux: [Achieving FIPS 140-3 validation](https://aws.amazon.com/blogs/compute/amazon-linux-2023-achieves-fips-140-3-validation/) You also can verify whether your operating system or cryptographic module is FIPS-validated using the [NIST database search tool](https://csrc.nist.gov/Projects/cryptographic-module-validation-program/validated-modules/search). ## FIPS validation of OpenSSL OpenSSL 3.0 and later versions introduced a FIPS provider that enables cryptographic operations in a FIPS-compliant mode. FIPS 140-3 validation: starting with OpenSSL 3.1.2, the library has [achieved FIPS 140-3 validation](https://openssl-library.org/post/2025-03-11-fips-140-3/) under certification [#4985](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4985). FIPS 140-2 validation: the FIPS provider for OpenSSL 3.0.x has been validated for FIPS 140-2 under certifications [#4811](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4811) and [#4282](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4282). ## Verification of correct operation of NGINX Plus The following process describes how to deploy NGINX Plus in a FIPS‑compliant environment and verify that the FIPS operations are functioning correctly. It involves three basic steps: - [Verify](#os-fips-check) if the operating system is running in FIPS mode. If not, [configure](#os-fips-setup) it to enable FIPS mode. - [Verify](#openssl-fips-check) that the OpenSSL library is operating in FIPS mode. - Run basic checks for [OpenSSL](#openssl-fips-check) and [NGINX Plus](#nginx-plus-fips-check) to confirm deployment in FIPS mode. The process uses Red Hat Enterprise Linux (RHEL) release 9.6 as an example and can be adapted for other Linux operating systems that can be configured in FIPS mode. ### Step 1: Configure the operating system to use FIPS mode {#os-fips-setup} For the purposes of the following demonstration, we installed and configured a RHEL 9.6 server. The [Red Hat FIPS documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/security_hardening/switching-rhel-to-fips-mode_security-hardening) explains how to switch the operating system between FIPS mode and non‑FIPS mode by editing the boot options and restarting the system. For instructions for enabling FIPS mode on other FIPS‑compliant Linux operating systems, see the operating system documentation, for example: - RHEL 9: [Switching to FIPS mode](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/security_hardening/switching-rhel-to-fips-mode_security-hardening) - Ubuntu: [Switching to FIPS mode](https://documentation.ubuntu.com/security/docs/compliance/fips/how-to-install-ubuntu-with-fips/) - SLES: [How to enable FIPS](https://www.suse.com/support/kb/doc/?id=000019432) - Oracle Linux 9: [Configuring FIPS mode](https://docs.oracle.com/en/operating-systems/oracle-linux/9/security/configuring_fips_mode.html#configuring-fips-mode) - Amazon Linux 2023: [Enabling FIPS mode](https://docs.aws.amazon.com/linux/al2023/ug/fips-mode.html) - Amazon Linux 2: [Enabling FIPS mode](https://docs.aws.amazon.com/linux/al2/ug/fips-mode.html) - AlmaLinux: [FIPS Validation for AlmaLinux](https://almalinux.org/blog/2023-09-19-fips-validation-for-almalinux/) ### Step 2: Verify the operating system is in FIPS mode {#os-fips-check} You can verify that the operating system is in FIPS mode and that the version of OpenSSL provided by the operating system vendor is FIPS‑compliant by using the following tests. **Check operating system flags**: When the operating system is in FIPS mode, `crypto.fips_enabled` kernlel flag is `1`; otherwise, it is `0`: ```shell sudo sysctl -a | grep fips ``` The output of the command shows that FIPS is enabled at the kernel level: ```none crypto.fips_enabled = 1 crypto.fips_name = Red Hat Enterprise Linux 9 - Kernel Cryptographic API crypto.fips_version = 5.14.0-570.39.1.el9_6.aarch64 ``` Check kernel logs for FIPS algorithm registration: ```shell journalctl -k -o cat -g alg: ``` The output of the command verifies the status of algorithm self-tests and whether certain algorithms are registered, passed FIPS self-tests, or are disabled due to FIPS mode being active: ```none alg: self-tests for pkcs1pad(rsa-generic,sha512) (pkcs1pad(rsa,sha512)) passed alg: self-tests for pkcs1pad(rsa-generic,sha256) (pkcs1pad(rsa,sha256)) passed alg: self-tests for cbc-aes-ce (cbc(aes)) passed alg: self-tests for jitterentropy_rng (jitterentropy_rng) passed alg: poly1305 (poly1305-neon) is disabled due to FIPS alg: xchacha12 (xchacha12-neon) is disabled due to FIPS alg: xchacha20 (xchacha20-neon) is disabled due to FIPS ``` Beyond kernel-level verification, you can ensure that the whole operating system environment is configured for FIPS compliance: ```shell sudo fips-mode-setup --check ``` The output of the command shows that FIPS is running: ```none FIPS mode is enabled. ``` ### Step 3: Verify the OpenSSL is in FIPS mode {#openssl-fips-check} **Determine the OpenSSL FIPS Provider is active**: This test verifies the correct version of OpenSSL and that the OpenSSL FIPS Provider is active: ```shell openssl list -providers | grep -A3 fips ``` The output of the command shows the FIPS provider status: ```none fips name: Red Hat Enterprise Linux 9 - OpenSSL FIPS Provider version: 3.0.7-395c1a240fbfffd8 status: active ``` **Determine whether OpenSSL can perform SHA1 hashes**: This test verifies the correct operation of OpenSSL. The SHA-1 hash algorithm, while considered weak, is still permitted in FIPS mode as it is included in FIPS-approved standards for certain legacy use cases. Failure of this command indicates that the OpenSSL implementation is not working properly: ```shell openssl sha1 /dev/null ``` The result of the command, showing the SHA1 checksum of `/dev/null`: ```none SHA1(/dev/null)= da39a3ee5e6b4b0d3255bfef95601890afd80709 ``` **Determine whether OpenSSL allows MD5 hashes**: This test verifies that OpenSSL is running in FIPS mode. MD5 is not a permitted hash algorithm in FIPS mode, so an attempt to use it fails: ```shell openssl md5 /dev/null ``` The result of the command: ```none Error setting digest 200458BAFFFF0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:355:Global default library context, Algorithm (MD5 : 95), Properties () 200458BAFFFF0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:272: ``` If OpenSSL is not running in FIPS mode, the MD5 hash functions normally: ```shell openssl md5 /dev/null ``` The result of the command, showing the MD5 checksum of `/dev/null`: ```none MD5(/dev/null)= d41d8cd98f00b204e9800998ecf8427e ``` ### Step 4: Install NGINX Plus on the operating system {#nginx-plus-instll} Follow the [F5 NGINX Plus Installation guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/) to install NGINX Plus on the host operating system, either directly from the [NGINX Plus repository](https://account.f5.com/myf5), or by downloading the **nginx-plus** package (**rpm** or **deb** package) onto another system and manually installing it on the host operating system. **Verify that NGINX Plus is correctly installed**: Run the following command to confirm that NGINX Plus is installed and is using the expected OpenSSL cryptographic module: ```shell nginx -V ``` Sample output from the command: ```shell nginx version: nginx/1.29.3 (nginx-plus-r36) built by gcc 11.5.0 20240719 (Red Hat 11.5.0-5) (GCC) built with OpenSSL 3.2.2 4 Jun 2024 ``` **Configure NGINX Plus to serve a simple SSL/TLS‑protected website**: Add the following simple configuration to NGINX Plus: ```nginx server { listen 443 ssl; ssl_certificate /etc/nginx/ssl/test.crt; ssl_certificate_key /etc/nginx/ssl/test.key; location / { root /usr/share/nginx/html; index index.html index.htm; } } ``` If necessary, you can generate a self‑signed certificate for test purposes: ```shell mkdir -p /etc/nginx/ssl && \ openssl req -newkey rsa:2048 -nodes -keyout /etc/nginx/ssl/test.key -x509 -days 365 -out /etc/nginx/ssl/test.crt ``` Verify that you can access the website using HTTPS from a remote host. Connect to the NGINX IP address using the `openssl s_client` command, and enter the HTTP message `GET /`: ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 ``` Use `openssl s_client` for this test because it unambiguously confirms which SSL/TLS cipher was negotiated in the connection. After some debugging information (including the cipher selected), the body of the default “Welcome to nginx!” greeting page is displayed. ### Step 5: Verify compliance with FIPS {#nginx-plus-fips-check} FIPS 140-2 and 140-3 disallows the use of some cryptographic algorithms, including the Camellia block cipher. In addition to FIPS 140-2, FIPS 140-3 disallows the use of several ciphers and algorithms that were once allowed or still allowed under FIPS 140-2. You can test compliance with FIPS 140-2 / 140-3 by issuing SSL/TLS requests with known ciphers on another (non-FIPS-mode) server: #### RC4-MD5 `RC4-MD5` is considered insecure and deprecated across all modern cryptographic standards. It is disallowed and disabled by default in FIPS-compliant OpenSSL, and TLS 1.2 and 1.3. The SSL handshake always fails. ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 -cipher RC4-MD5 ``` For FIPS compliance, alternative cipher suites can be used such as: - `TLS_RSA_WITH_AES_128_GCM_SHA256` - `TLS_RSA_WITH_AES_256_GCM_SHA384` - `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256` - `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` #### CAMELLIA-SHA ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 -cipher CAMELLIA256-SHA ``` This cipher is considered secure but is not permitted by the FIPS standard. The SSL handshake fails if the target system is compliant with FIPS 140-2 /140-3, and succeeds otherwise. #### AES256-SHA The cipher is permitted under FIPS 140-2 as it combines AES encryption with SHA-1. However, under FIPS 140-3, SHA-1 is explicitly disallowed due to its vulnerabilities, such as susceptibility to collision attacks. As a result, the SSL handshake fails under FIPS 140-3 and succeeds under FIPS 140-2: ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 -cipher AES256-SHA ``` For FIPS 140-3 compliance, alternative cipher suites that leverage SHA-2 or SHA-3 for hashing can be used: - AES-GCM-Based Cipher Suites (TLS 1.2): - `TLS_RSA_WITH_AES_256_GCM_SHA384` - `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` - ChaCha20-Based Cipher Suites (TLS 1.2 or 1.3): - `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` - TLS 1.3 Cipher Suites: - `TLS_AES_256_GCM_SHA384` - `TLS_AES_128_GCM_SHA256` - `TLS_CHACHA20_POLY1305_SHA256` #### 3DES The `3DES` (Triple DES) cipher is allowed under FIPS 140-2, but disallowed under FIPS 140-3. NIST deprecated its use [starting January 1, 2024](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar2.pdf) due to its reduced security strength (112 bits) and vulnerability to brute-force attacks. As a result, the SSL handshake always fails in FIPS-3 compliant environment: ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 -cipher DES-CBC3 ``` For FIPS 140-3 compliance, AES-Based or ChaCha20-Based cipher suites can be used: - `TLS_RSA_WITH_AES_128_GCM_SHA256` - `TLS_RSA_WITH_AES_256_GCM_SHA384` - `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256` - `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` - `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` #### DH and DSA Under FIPS 140-2, Diffie-Hellman (DH) and Digital Signature Algorithm (DSA) were permitted with a minimum key size of 1024 bits. However, under FIPS 140-3, the minimum key size for both DH and DSA has been increased to 2048 bits. For example, the `TLS_DH_RSA_WITH_AES_128_CBC_SHA` algorithm is FIPS 140-2 compliant, but not FIPS 140-3 compliant due to its use of DH with a key size of less than 2048 bits, CBC mode encryption, and SHA-1 hashing: ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 -cipher TLS_DH_RSA_WITH_AES_128_CBC_SHA ``` The `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256` algorithm is FIPS 140-3 compliant as it uses Elliptic Curve Diffie-Hellman Ephemeral (ECDHE), AES-GCM for encryption, and SHA-256 for hashing: ```shell (echo "GET /" ; sleep 1) | openssl s_client -connect :443 -cipher TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 ``` ## Ciphers disabled in FIPS Mode The FIPS 140-2 standard only permits a [subset of the typical SSL and TLS ciphers](https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/fips1402annexa.pdf), while FIPS 140-3 [extends this requirements](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf) to enforce stricter cryptographic algorithms. In the following test, the ciphers presented by NGINX Plus are surveyed using the `nmap` utility (installed separately). In its default configuration, with the [`ssl_ciphers HIGH:!aNULL:!MD5`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive, NGINX Plus presents the following ciphers to SSL/TLS clients: ```shell nmap --script ssl-enum-ciphers -p 443 ``` The output of the command for NGINX Plus running on Red Hat Enterprise Linux 9 without FIPS enabled: ```shell PORT STATE SERVICE 443/tcp open https | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CCM (rsa 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CCM (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_ARIA_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_ARIA_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 (rsa 2048) - A | compressors: | NULL | cipher preference: client | TLSv1.3: | ciphers: | TLS_AKE_WITH_AES_128_CCM_SHA256 (ecdh_x25519) - A | TLS_AKE_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A | TLS_AKE_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A | TLS_AKE_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A | cipher preference: client |_ least strength: A ``` When FIPS 140-3 mode is enabled, NGINX Plus presents the following ciphers to SSL/TLS clients: ```shell PORT STATE SERVICE 443/tcp open https | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A | compressors: | NULL | cipher preference: client | TLSv1.3: | ciphers: | TLS_AKE_WITH_AES_128_CCM_SHA256 (secp256r1) - A | TLS_AKE_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_AKE_WITH_AES_256_GCM_SHA384 (secp256r1) - A | cipher preference: client |_ least strength: A ``` Based on the results above, the following ciphers are disallowed under FIPS 140-3 compliance: - Camellia-Based Ciphers: FIPS compliance requires cryptographic algorithms to be validated by NIST, and Camellia is not NIST-approved despite being recognized by ISO/IEC standards. - `TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256` - `TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384` - `TLS_RSA_WITH_CAMELLIA_128_CBC_SHA` - `TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256` - `TLS_RSA_WITH_CAMELLIA_256_CBC_SHA` - `TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256` - ARIA-Based Ciphers: similar to Camellia, ARIA is not a NIST-approved algorithm and is therefore excluded from FIPS compliance. - `TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256` - `TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384` - `TLS_RSA_WITH_ARIA_128_GCM_SHA256` - `TLS_RSA_WITH_ARIA_256_GCM_SHA384` - RSA Key Exchange Ciphers: static RSA key exchange lacks Forward Secrecy, allowing decryption of past traffic if the private key is compromised, thus disallowed in FIPS mode. - `TLS_RSA_WITH_AES_128_CBC_SHA` - `TLS_RSA_WITH_AES_128_CBC_SHA256` - `TLS_RSA_WITH_AES_128_GCM_SHA256` - `TLS_RSA_WITH_AES_256_CBC_SHA` - `TLS_RSA_WITH_AES_256_CBC_SHA256` - `TLS_RSA_WITH_AES_256_GCM_SHA384` - CBC Mode Ciphers (Non-AEAD: CBC is vulnerable to padding oracle attacks (e.g., POODLE, Lucky13), making it insecure. FIPS 140-3 prioritizes AEAD modes like AES-GCM and AES-CCM. - `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA` - `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256` - `TLS_RSA_WITH_AES_128_CBC_SHA` - `TLS_RSA_WITH_AES_128_CBC_SHA256` - `TLS_RSA_WITH_AES_256_CBC_SHA` - `TLS_RSA_WITH_AES_256_CBC_SHA256` - ChaCha20-Poly1305: it is not a NIST-approved algorithm and is excluded from FIPS compliance. FIPS exclusively permits algorithms such as `AES-GCM` and `AES-CCM`. - `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` - AES-CCM Variants: - `TLS_RSA_WITH_AES_128_CCM` - `TLS_RSA_WITH_AES_256_CCM` You can also use the [Qualys SSL server test](https://www.ssllabs.com/ssltest) to verify the ciphers presented by NGINX Plus to SSL/TLS clients. ## Conclusion NGINX Plus can be used to decrypt and encrypt SSL/TLS‑encrypted network traffic in deployments that require FIPS 140-2 Level 1 or FIPS 140-3 Level 1 compliance. The process described above may be used to verify that NGINX Plus is operating in conformance with the FIPS 140-2 Level 1 and FIPS 140-3 Level 1 standards. ## Definition of terms - **Cryptographic module**: The OpenSSL software, comprised of libraries of FIPS‑validated algorithms that can be used by other applications. - **Cryptographic boundary**: The operational functions that use FIPS‑validated algorithms. For NGINX Plus, the cryptographic boundary includes all functionality that is implemented by the [`http_auth_jwt`](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html), [`http_ssl`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html), [`http_v2`](https://nginx.org/en/docs/http/ngx_http_v2_module.html), [`http_v3`](https://nginx.org/en/docs/http/ngx_http_v3_module.html), [`mail_ssl`](https://nginx.org/en/docs/mail/ngx_mail_ssl_module.html), and [`stream_ssl`](https://nginx.org/en/docs/stream/ngx_stream_ssl_module.html) modules. These modules implement SSL and TLS operations for inbound and outbound connections which use HTTP, HTTP/2, HTTP/3, TCP, and mail protocols. - **NGINX Plus**: The NGINX Plus software application developed by F5, Inc. and delivered in binary format from F5 servers. - **FIPS mode**: When the operating system is configured to run in FIPS mode, the OpenSSL cryptographic module operates in a mode that has been validated to be in compliance with FIPS 140-2 Level 1 or FIPS 140-3 Level 1. Most operating systems do not run in FIPS mode by default, so explicit configuration is necessary to enable FIPS mode. - **FIPS validated**: A component of the OpenSSL cryptographic module (the OpenSSL FIPS Object Module) is formally validated by an authorized certification laboratory. The validation holds if the module is built from source with no modifications to the source or build process. The implementation of FIPS mode that is present in operating system vendors’ distributions of OpenSSL contains this validated module. - **FIPS compliant**: NGINX Plus is compliant with FIPS 140-2 Level 1 and FIPS 140-3 Level 1 within the cryptographic boundary when used with a FIPS‑validated OpenSSL cryptographic module on an operating system running in FIPS mode. ## See also [FIPS 140-3 Standard in the PDF format](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf) [FIPS compliance with NGINX Plus and Red Hat Enterprise Linux](https://www.f5.com/pdf/technology-alliances/fips-compliance-made-simple-with-f5-and-red-hat-one-pager.pdf) [F5 NGINX Plus running on Red Hat Enterprise Linux is now FIPS 140-3 compliant](https://www.redhat.com/en/blog/f5-nginx-plus-running-red-hat-enterprise-linux-now-fips-140-3-compliant) ## Revision history - Version 2 (September 2025) - Added information about FIPS 140-3 Level 1 compliance, updated test procedures, updated product versions, updated URLs to operating systems, NIST certificates and other relevant resources. - Version 1 (August 2019) - Initial version with FIPS 140-2 Level 1 compliance. -------------------------------------------------------------------------------- title: "Single Sign-On with OneLogin" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/onelogin/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using OneLogin as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [OneLogin](https://www.onelogin.com/) as the Identity Provider (IdP) and NGINX Plus as the Relying Party (RP), or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. In the latest NGINX Plus version, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - An [OneLogin](https://www.onelogin.com/) account with administrator privileges. - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure OneLogin {#onelogin-setup} ### Create a OneLogin OIDC Application 1. Log in to your OneLogin admin console, for example, `https://.onelogin.com`. 2. In the navigation bar, select **Applications**. 3. Click the **Add App** button. - On the **Find Applications** page, search for **OpenID Connect (OIDC)** and then select it. - Enter a **Display Name**, for example, `NGINX Demo App`. - Select **Save**. 4. In the app navigation, select **Configuration**. - In **Redirect URIs**, add the callback URI for your NGINX Plus instance, for example, `https://demo.example.com/oidc_callback`. - In **Logout URL**, add the post logout redirect URI, for example, `https://demo.example.com/post_logout/`. - Select **Save**. 5. In the app navigation, select **SSO**. - Copy the **Client ID**. You will need it later when configuring NGINX Plus. - Select **Show client secret** and copy the **Client Secret**. You will need it later when configuring NGINX Plus. - Copy the **Issuer** URL, or OpenID Connect Discovery URL. You will need it later when configuring NGINX Plus. For OneLogin, the Issuer ID generally structured as: `https://.onelogin.com/oidc/2` See [Provider Configuration](https://developers.onelogin.com/openid-connect/api/provider-config) for details. ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, OneLogin publishes the `.well-known/openid-configuration` document at the following address: `https://.onelogin.com/oidc/2/.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https://.onelogin.com/oidc/2/.well-known/openid-configuration | jq ``` Where: - the `.onelogin.com` is your OneLogin subdomain - the `/oidc/2` is the OneLogin OIDC endpoint version - the `/.well-known/openid-configuration` is the default address for OneLogin for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://.onelogin.com/oidc/2", "authorization_endpoint": "https://.onelogin.com/oidc/2/auth", "token_endpoint": "https://.onelogin.com/oidc/2/token", "jwks_uri": "https://.onelogin.com/oidc/2/certs", "userinfo_endpoint": "https://.onelogin.com/oidc/2/me", "end_session_endpoint": "https://.onelogin.com/oidc/2/logout", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for OneLogin is `https://.onelogin.com/oidc/2`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ### Assign Users and Groups 1. In the app navigation, select **Users** > **Roles**. 2. Add users and groups who should have access to this application. ## Set up NGINX Plus {#nginx-plus-setup} With Onelogin configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [Onelogin Configuration](#onelogin-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the OneLogin provider named `onelogin` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider onelogin { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - your actual OneLogin **Client ID** obtained in [OneLogin Configuration](#onelogin-setup) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - your **Client Secret** obtained in [OneLogin Configuration](#onelogin-setup) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - the **Issuer** URL obtained in [OneLogin Configuration](#onelogin-setup) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your OneLogin OIDC URL: `https://.onelogin.com/oidc/2`. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where OneLogin should redirect the user after a successful logout. This value **must also be configured** in the OneLogin application's Logout URL setting. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user's ID token as a *hint* to OneLogin. This directive is **required** by OneLogin when `post_logout_redirect_uri` is used. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch `/oidc/2/me` from the OneLogin and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when OneLogin's OpenID Connect discovery document advertises the `S256` code challenge method in the `code_challenge_methods_supported` field. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP metadata does not list `S256`. - The module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`). - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider onelogin { issuer https://.onelogin.com/oidc/2; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with OneLogin OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `onelogin` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#onelogin-setup-oidc-provider): ```nginx # ... location / { auth_oidc onelogin; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by OneLogin: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by OneLogin - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc onelogin; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nOneLogin sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider onelogin { # The 'issuer' is typically your OneLogin OIDC base URL # e.g. https://.onelogin.com/oidc/2 issuer https://.onelogin.com/oidc/2; # Replace with your actual OneLogin Client ID and Secret client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Protect this path with OneLogin OIDC auth_oidc onelogin; # Forward OIDC claims to the upstream as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nOneLogin sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open `https://demo.example.com/` in a browser. You will be automatically redirected to the OneLogin sign-in page. 2. Enter valid OneLogin credentials of a user who has access the application. Upon successful sign-in, OneLogin redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; OneLogin ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to OneLogin for a fresh sign‑in, proving the session has been terminated. **Note:** If you restricted access to a group of users, be sure to select a user who has access to the application. ## Legacy njs-based OneLogin Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based OneLogin Guide](nginx/deployment-guides/single-sign-on/oidc-njs/onelogin.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added PKCE configuration (`pkce` directive) and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "Dynamic Configuration of Upstreams with the NGINX Plus API" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/dynamic-configuration-api/index.md description: Dynamically reconfigure the servers in an F5 NGINX Plus upstream group using the NGINX Plus API, without reloading configuration or restarting processes. -------------------------------------------------------------------------------- ## Overview With F5 NGINX Plus, configuration of upstream servers in a server group can be modified on-the-fly without reloading the servers and NGINX configuration. This is useful for: - autoscaling, when you need to add more servers - maintenance, when you need to remove a server, specify a backup server, or take a server down temporarily - quick setup, when you need to change upstream server settings such as server weight, active connections, slow start, failure timeouts. - monitoring, when you get the state of the server or server group with one command These changes are made with the NGINX Plus REST API interface with API commands. > **Note:** In NGINX Plus [Release 12](nginx/releases.md#r12) and earlier, dynamic configuration was performed with the `upstream_conf` handler. That API (and the extended `status` API) are now deprecated in favor of the NGINX Plus API. ## Prerequisites Prior to using the dynamic configuration feature, make sure that you have the following environment: 1. NGINX Plus [R13](nginx/releases.md#r13) or later 2. You have created upstream groups of application or web servers, as described in [HTTP Load Balancing](http-load-balancer.md) and [TCP/UDP Load Balancing](tcp-udp-load-balancer.md) 3. Upstream server groups reside in the shared memory zone, as described in [Sharing Data with Multiple Worker Processes](http-load-balancer.md) ## Enabling Dynamic Configuration 1. Create an upstream server group as described in Proxying Traffic to a Group of Servers. ```nginx http { # ... upstream appservers { server appserv1.example.com weight=5; server appserv2.example.com:8080 fail_timeout=5s; server reserve1.example.com:8080 backup; server reserve2.example.com:8080 backup; } server { # Location that proxies requests to the upstream group location / { proxy_pass http://appservers; health_check; } } } ``` 1. Include the [`zone`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) directive in the `upstream` block. The `zone` directive configures a zone in the shared memory and sets the zone name and size. The configuration of the server group is kept in this zone, so all worker processes use the same configuration: ```nginx http { # ... upstream appservers { zone appservers 64k; server appserv1.example.com weight=5; server appserv2.example.com:8080 fail_timeout=5s; server reserve1.example.com:8080 backup; server reserve2.example.com:8080 backup; } } ``` 2. Enable the NGINX API in read‑write mode by including the [`api`](https://nginx.org/en/docs/http/ngx_http_api_module.html#api) directive in a dedicated [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) block in a [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) block. We strongly recommend restricting access to the location and to `PATCH`/`POST`/`DELETE` methods. This example uses the [`allow`](https://nginx.org/en/docs/http/ngx_http_access_module.html#allow) and [`deny`](https://nginx.org/en/docs/http/ngx_http_access_module.html#deny) directives to grant access from the `localhost` address (`127.0.0.1`) and deny access from all other addresses. It also restricts access to `PATCH`/`POST`/`DELETE` methods with [HTTP basic authentication](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html): ```nginx server { location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } api write=on; allow 127.0.0.1; deny all; } } ``` Complete example: ```nginx http { # ... # Configuration of the server group upstream appservers { zone appservers 64k; server appserv1.example.com weight=5; server appserv2.example.com:8080 fail_timeout=5s; server reserve1.example.com:8080 backup; server reserve2.example.com:8080 backup; } server { # Location that proxies requests to the upstream group location / { proxy_pass http://appservers; health_check; } # Location for dynamic configuration requests location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } api write=on; allow 127.0.0.1; deny all; } } } ``` ## Using the API for Dynamic Configuration The NGINX Plus REST API supports the following HTTP methods: - `GET` – Display information about an upstream group or individual server in it - `POST` – Add a server to the upstream group - `PATCH` – Modify the parameters of a particular server - `DELETE` – Delete a server from the upstream group The endpoints and methods for the NGINX Plus API are described in the [NGINX Modules Reference](https://nginx.org/en/docs/http/ngx_http_api_module.html). In addition, the API has a built‑in a Swagger specification that can be used to explore the API and understand the capabilities of each resource. The Swagger documentation can be accessed at `http://_NGINX-host_/swagger-ui/`. To change the configuration of an upstream group dynamically, send an HTTP request with the appropriate API method. The following examples use the `curl` command, but any mechanism for making HTTP requests is supported. All request bodies and responses are in JSON format. The URI specifies the following information in this order: - The hostname or IP address of the node that handles the request (in the following examples, `127.0.0.1`) - The location where the `api` directive appears (`api`) - The API version (`9`) - The name of the upstream group, complete its place in the NGINX Plus configuration hierarchy represented as a slash‑separated path (`http/upstreams/appservers`) For example, to add a new server to the `appservers` upstream group, send the following `curl` command: ```shell curl -X POST -d '{ \ "server": "10.0.0.1:8089", \ "weight": 4, \ "max_conns": 0, \ "max_fails": 0, \ "fail_timeout": "10s", \ "slow_start": "10s", \ "backup": true, \ "down": true \ }' -s 'http://127.0.0.1/api/9/http/upstreams/appservers/servers' ``` To remove a server from the upstream group: ```shell curl -X DELETE -s 'http://127.0.0.1/api/9/http/upstreams/appservers/servers/0' ``` To set the `down` parameter for the first server in the group (with ID `0`): ```shell curl -X PATCH -d '{ "down": true }' -s 'http://127.0.0.1/api/9/http/upstreams/appservers/servers/0' ``` ### Interactive Example You can explore the Swagger interface to the NGINX Plus API in read‑only mode at [https://demo.nginx.com/swagger-ui/](https://demo.nginx.com/swagger-ui/). ## Configuring Persistence of Dynamic Configuration With the basic configuration in [Enabling the API](#api_setup), changes made with the API are stored only in the shared memory zone. The changes are discarded when the NGINX Plus configuration file is reloaded. To make the changes persist across configuration reloads, move the list of upstream servers from the `upstream` block to a special file for storing server state, defined with the [`state`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#state) directive. The recommended path for Linux distributions is `/var/lib/nginx/state/`, and for FreeBSD distributions is `/var/db/nginx/state/`. ```nginx http { # ... upstream appservers { zone appservers 64k; state /var/lib/nginx/state/appservers.conf; # All servers are defined in the state file # server appserv1.example.com weight=5; # server appserv2.example.com:8080 fail_timeout=5s; # server reserve1.example.com:8080 backup; # server reserve2.example.com:8080 backup; } } ``` Keep in mind that the state file can be modified only with configuration commands from the [API](https://nginx.org/en/docs/http/ngx_http_api_module.html) interface; do not modify the file directly (for example, using a text editor). -------------------------------------------------------------------------------- title: "NGINX Directives Index" url: https://docs.nginx.com/nginx/directives/index.md -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Restricting Access to Proxied TCP Resources" url: https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-tcp/index.md description: Protect your upstream TCP application servers by limiting connections or bandwidth, based on client IP address or other variables. -------------------------------------------------------------------------------- This chapter provides scenarios for restricting access to a database or media server that communicates over TCP. Access can be limited by IP address, the number of simultaneous connections, or bandwidth. ## Restricting Access by IP Address NGINX can allow or deny access based on a particular IP address or the range of IP addresses of client computers. To allow or deny access, use the [allow](https://nginx.org/en/docs/stream/ngx_stream_access_module.html#allow) and [deny](https://nginx.org/en/docs/stream/ngx_stream_access_module.html#deny) directives inside the [stream](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context or a [server](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#server) block: ```nginx stream { #... server { listen 12345; deny 192.168.1.2; allow 192.168.1.1/24; allow 2001:0db8::/32; deny all; } } ``` The rules are processed in sequence, from top to bottom: if the first directive in the sequence is `deny all`, then all further `allow` directives have no effect. In this example, the subnet `192.168.1.1/24` is allowed access, with the exception of `192.168.1.2`. The `2001:0db8::/32` range of IPv6 addresses is also allowed, and access to any other IP addresses is denied. ## Limiting the Number of TCP Connections You can limit the number of simultaneous TCP connections from one IP address. This can be useful in preventing denial-of-service (DoS) attacks. First, let’s define the _zone_ that will store the maximum number of TCP connections to one server, and a key to identify the connection. This can be done with the [limit_conn_zone](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn_zone) directive in the [stream](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context: ```nginx stream { #... limit_conn_zone $binary_remote_addr zone=ip_addr:10m; #... } ``` The key that identifies the connection is defined as `$binary_remote_addr`, which represents the IP address of the client in binary format. The name of the shared memory zone is `ip_addr` and the zone size is 10 megabytes. After the zone is defined, limit connections with the [limit_conn](https://nginx.org/en/docs/stream/ngx_stream_limit_conn_module.html#limit_conn) directive. Its first parameter specifies the name of the shared memory zone previously defined by [limit_conn_zone](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn_zone). As the second parameter, specify the maximum number of allowed connections for each IP address, in either the [stream](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) context or a [server](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#server) block (as in this example, which also shows the prerequisite `limit_conn_zone` directive): ```nginx stream { #... limit_conn_zone $binary_remote_addr zone=ip_addr:10m; server { #... limit_conn ip_addr 1; } } ``` When limiting the number of connections per IP address, be aware that multiple hosts behind a Network Address Translation (NAT) device share the same IP address. ## Limiting the Bandwidth You can configure the maximum download or upload speed for TCP connections. Include the [proxy_download_rate](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_download_rate) or [proxy_upload_rate](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_upload_rate) directive, respectively: ```nginx server { #... proxy_download_rate 100k; proxy_upload_rate 50k; } ``` With these settings a client can download data through a single connection at a maximum speed of 100 kilobytes per second, and upload data through a single connection at a maximum speed of 50 kilobytes per second. However, the client can open several connections. So if the goal is to limit overall speed of loading for each client, the number of connections must also be limited to `1` as described in the previous section. ```nginx stream { #... limit_conn_zone $binary_remote_addr zone=ip_addr:10m; server { #... limit_conn ip_addr 1; proxy_download_rate 100k; proxy_upload_rate 50k; } } ``` -------------------------------------------------------------------------------- title: "Single Sign-On with Okta" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/okta/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Okta as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Okta](https://www.okta.com/) as the Identity Provider (IdP), and NGINX Plus as the Relying Party, or OIDC client application that verifies user identity. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. Starting from NGINX Plus version R34, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - An [Okta](https://www.okta.com/) administrator account with privileges to create and manage applications. - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure Okta {#okta-setup} In Okta, register a new application for NGINX Plus as the OIDC client to obtain the Client ID, Client Secret, and required OIDC endpoints. 1. Log in to your Okta admin console. 2. In the Admin Console, go to **Applications** > **Applications**. 3. Select **Create App Integration**. 4. In **Create a new app integration**, select: - **Sign-in method**: `OIDC - OpenID Connect`. - **Application type**: `Web Application`. - Select **Next**. 5. In **New Web App Integration**: - Enter the **Name** for your new application, for example, **Nginx Demo App**. - Add a URI for the OIDC callback in **Sign-in redirect URIs**, for example, `https://demo.example.com/oidc_callback`. - Add a URI for post logout redirect in **Sign-out redirect URIs**, for example, `https://demo.example.com/post_logout/`. - Select **Save**. 6. In **Applications**, select **Nginx Demo App**. 7. In the **General** tab: - Copy the **Client ID**. You will need it later when configuring NGINX Plus. - Copy the **Client secret**. You will need it later when configuring NGINX Plus. 8. In the **Sign On** tab: - Copy the **Okta Issuer (Authorization Server)**, for example: `https://dev-123456.oktapreview.com/oauth2/default` You will need it later when configuring NGINX Plus. ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, Okta publishes the `.well-known/openid-configuration` document at the following address: `https:///oauth2/default/.well-known/openid-configuration`. 1. Run the following `curl` command in a terminal: ```shell curl https://dev-123456.okta.com/oauth2/default/.well-known/openid-configuration | jq ``` Where: - the `dev-123456.okta.com` is your Okta domain - the `/oauth2/default` is the default authorization server - the `/.well-known/openid-configuration` is the default address for Okta for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://dev-123456.okta.com/oauth2/default", "authorization_endpoint": "https://dev-123456.okta.com/oauth2/default/v1/authorize", "token_endpoint": "https://dev-123456.okta.com/oauth2/default/v1/token", "jwks_uri": "https://dev-123456.okta.com/oauth2/default/v1/keys", "userinfo_endpoint": "https://dev-123456.okta.com/oauth2/default/v1/userinfo", "end_session_endpoint": "https://dev-123456.okta.com/oauth2/default/v1/logout", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for Okta is `https:///oauth2/default`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ### Assign Users or Groups By default, Okta might limit application access to certain users or groups. To add or remove users in Okta: 1. Log in to your Okta admin console. 2. In **Applications**, choose **Nginx Demo App**. 3. Go to **Assignments**. 4. Add or remove users and groups that can access this application. ### Enable Front-Channel Single Logout (optional) {#okta-frontchannel-logout} Front-channel logout allows Okta to notify NGINX Plus when a user signs out of other Single Logout (SLO)-participating applications or from the Okta dashboard itself. Okta sends a front-channel HTTP request (typically loaded in a hidden iframe) to a logout URL that you configure in the application settings. This feature is optional and requires NGINX Plus Release 36 or later, together with the `frontchannel_logout_uri` directive in the NGINX OIDC provider configuration. 1. In the Okta Admin Console, go to **Settings** > **Features**. 2. Locate **Front-channel Single Logout** and enable it for your org. 3. Go to **Applications** > **Applications** and open your **Nginx Demo App** (or the OIDC app that you created earlier). 4. On the **General** tab, click **Edit**. 5. In the **Logout** section, under **Single Logout**, enable **User logs out of other logout-initiating apps or Okta**. This option enables front-channel SLO for the app. 6. In **Logout request URL**, enter the NGINX Plus front-channel logout endpoint, for example: ```text https://demo.example.com/front_logout ``` 7. Select **Include user session details**. With this option enabled, Okta includes the user session identifier (sid) and issuer (iss) in the front-channel logout request that it sends to NGINX Plus. 8. Click **Save**. ## Set up NGINX Plus {#nginx-plus-setup} With Okta configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [Okta Configuration](#okta-setup). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the Okta provider named `okta` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider okta { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - your actual Okta **Client ID** obtained in [Okta Configuration](#okta-setup) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - your **Client Secret** obtained in [Okta Configuration](#okta-setup) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - the **Issuer** URL obtained in [Okta Configuration](#okta-setup) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your Okta OIDC URL: `https://dev-123456.okta.com/oauth2/default`. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where Okta should redirect the user after a successful logout. This value **must also be configured** in the Okta application's Sign-out redirect URIs. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user's ID token as a *hint* to Okta. This directive is **optional**, however, if it is omitted the Okta may display an extra confirmation page asking the user to approve the logout request. - The **frontchannel_logout_uri** directive defines the URI that receives OpenID Connect front-channel logout requests from Okta. This URI must be an HTTPS path hosted by NGINX Plus and must match the *Front-channel logout URL* configured in the Okta application registration. When a front-channel logout GET request is received at this URI (typically in a hidden iframe), the OIDC module clears the local session for the affected user. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch `/oauth2/default/v1/userinfo` from the Okta and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when the provider metadata advertises the `S256` code challenge method in the `code_challenge_methods_supported` field of the discovery document. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP metadata does not list `S256`. - The module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`). - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider okta { issuer https://dev-123456.okta.com/oauth2/default; client_id ; client_secret ; logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; frontchannel_logout_uri /front_logout; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with Okta OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `okta` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#okta-setup-oidc-provider): ```nginx # ... location / { auth_oidc okta; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by Okta: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by Okta - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable ```nginx # ... location / { auth_oidc okta; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nOkta sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; # Define the OIDC provider block for Okta oidc_provider okta { # The 'issuer' is your Okta issuer URL # For okta dev it looks like: https://dev-123456.okta.com/oauth2/default issuer https://dev-123456.okta.com/oauth2/default; # Your Okta "Client ID" from the application settings client_id ; # Your Okta "Client Secret" from the application settings client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Front-channel logout (OP‑initiated single sign-out) frontchannel_logout_uri /front_logout; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Enforce OIDC authentication with Okta auth_oidc okta; # Pass OIDC claims as HTTP headers to the backend proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nOkta sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open `https://demo.example.com/` in a browser. You will be automatically redirected to the Okta sign-in page. 2. Enter valid Okta credentials of a user who has access the application. Upon successful sign-in, Okta redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; Okta ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to Okta for a fresh sign‑in, proving the session has been terminated. **Note:** If you restricted access to a group of users, be sure to select a user who has access to the application. ## Legacy njs-based Okta Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based Okta Guide](nginx/deployment-guides/single-sign-on/oidc-njs/okta.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added front-channel logout support (`frontchannel_logout_uri`), PKCE configuration (`pkce` directive), and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version (NGINX Plus Release 34). -------------------------------------------------------------------------------- title: "Accepting the PROXY Protocol" url: https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/index.md -------------------------------------------------------------------------------- This article explains how to configure NGINX and F5 NGINX Plus to accept the PROXY protocol, rewrite the IP address of a load balancer or proxy to the one received in the PROXY protocol header, configure simple logging of a client’s IP address, and enable the PROXY protocol between NGINX and a TCP upstream server. ## Introduction The [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) enables NGINX and NGINX Plus to receive client connection information passed through proxy servers and load balancers such as HAproxy and Amazon Elastic Load Balancer (ELB). With the PROXY protocol, NGINX can learn the originating IP address from HTTP, SSL, HTTP/2, SPDY, WebSocket, and TCP. Knowing the originating IP address of a client may be useful for setting a particular language for a website, keeping a denylist of IP addresses, or simply for logging and statistics purposes. The information passed via the PROXY protocol is the client IP address, the proxy server IP address, and both port numbers. Using this data, NGINX can get the originating IP address of the client in several ways: - With the [`$proxy_protocol_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_addr) and [`$proxy_protocol_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_port) variables which capture the original client IP address and port. The [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr) and [`$remote_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_port) variables capture the IP address and port of the load balancer. - With the [RealIP](https://nginx.org/en/docs/http/ngx_http_realip_module.html) module which rewrites the values in the [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr) and [`$remote_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_port) variables, replacing the IP address and port of the load balancer with the original client IP address and port. The [`$realip_remote_addr`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#var_realip_remote_addr) and [`$realip_remote_port`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#var_realip_remote_port) variables retain the address and port of the load balancer, and the [`$proxy_protocol_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_addr) and [`$proxy_protocol_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_port) variables retain the original client IP address and port anyway. ## Prerequisites - To accept the PROXY protocol v2, NGINX Plus [R16](nginx/releases.md#r16) and later or NGINX Open Source [1.13.11](https://nginx.org/en/CHANGES) and later - To accept the PROXY protocol for HTTP, NGINX Plus [R3](nginx/releases.md#r3) and later or NGINX Open Source [1.5.12](https://nginx.org/en/CHANGES) and later - For TCP client‑side PROXY protocol support, NGINX Plus [R7](nginx/releases.md#r7) and later or NGINX Open Source [1.9.3](https://nginx.org/en/CHANGES) and later - To accept the PROXY protocol for TCP, NGINX Plus [R11](nginx/releases.md#r11) and later or NGINX Open Source [1.11.4](https://nginx.org/en/CHANGES) and later - The Real‑IP modules for [HTTP](https://nginx.org/en/docs/http/ngx_http_realip_module.html) and [Stream TCP](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html) are not included in NGINX Open Source by default; see [Installing NGINX Open Source](nginx/admin-guide/installing-nginx/installing-nginx-open-source.md) for details. No extra steps are required for NGINX Plus. ## Configure NGINX to Accept the PROXY Protocol To configure NGINX to accept PROXY protocol headers, add the `proxy_protocol` parameter to the `listen` directive in a `server` block in the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#listen) or [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#listen) block. ```nginx http { #... server { listen 80 proxy_protocol; listen 443 ssl proxy_protocol; #... } } stream { #... server { listen 12345 proxy_protocol; #... } } ``` Now you can use the [`$proxy_protocol_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_addr) and [`$proxy_protocol_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_port) variables for the client IP address and port and additionally configure the [HTTP](https://nginx.org/en/docs/http/ngx_http_realip_module.html) and [`stream`](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html) RealIP modules to replace the IP address of the load balancer in the [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr) and [`$remote_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_port) variables with the IP address and port of the client. ## Change the Load Balancer's IP Address To the Client IP Address You can replace the address of the load balancer or TCP proxy with the client IP address received from the PROXY protocol. This can be done with the [HTTP](https://nginx.org/en/docs/http/ngx_http_realip_module.html) and [`stream`](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html) RealIP modules. With these modules, the [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr) and [`$remote_port`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_port) variables retain the real IP address and port of the client, while the [`$realip_remote_addr`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#var_realip_remote_addr) and [`$realip_remote_port`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#var_realip_remote_port) variables retain the IP address and port of the load balancer. To change the IP address from the load balancer's IP address to the client's IP address: 1. Make sure you've configured NGINX to accept the PROXY protocol headers. See [Configuring NGINX to Accept the PROXY Protocol](#listen). 2. Make sure that your NGINX installation includes the [HTTP](https://nginx.org/en/docs/http/ngx_http_realip_module.html) and [Stream](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html) Real‑IP modules: ```shell nginx -V 2>&1 | grep -- 'http_realip_module' nginx -V 2>&1 | grep -- 'stream_realip_module' ``` If not, recompile NGINX with these modules. See [Installing NGINX Open Source](nginx/admin-guide/installing-nginx/installing-nginx-open-source.md) for details. No extra steps are required for NGINX Plus. 3. In the `set_real_ip_from` directive for [HTTP](https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from), [Stream](https://nginx.org/en/docs/stream/ngx_stream_realip_module.html#set_real_ip_from), or both, specify the IP address or the CIDR range of addresses of the TCP proxy or load balancer: ```nginx server { #... set_real_ip_from 192.168.1.0/24; #... } ``` 4. In the `http {}` context, change the IP address of the load balancer to the IP address of the client received from the PROXY protocol header, by specifying the `proxy_protocol` parameter to the [`real_ip_header`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header) directive: ```nginx http { server { #... real_ip_header proxy_protocol; } } ``` ## Log the Original IP Address When you know the original IP address of the client, you can configure the correct logging: 1. For HTTP, configure NGINX to pass the client IP address to upstream servers using the [`$proxy_protocol_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_addr) variable with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive: ```nginx http { proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-For $proxy_protocol_addr; } ``` 2. Add the [`$proxy_protocol_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_addr) variable to the `log_format` directive ([HTTP](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) or [Stream](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format)): - In the `http` block: ```nginx http { #... log_format combined '$proxy_protocol_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; } ``` - In the `stream` block: ```nginx stream { #... log_format basic '$proxy_protocol_addr - $remote_user [$time_local] ' '$protocol $status $bytes_sent $bytes_received ' '$session_time'; } ``` ## PROXY Protocol for a TCP Connection to an Upstream For a TCP stream, the PROXY protocol can be enabled for connections between NGINX and an upstream server. To enable the PROXY protocol, include the [`proxy_protocol`](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol) directive in a `server` block at the `stream {}` level: ```nginx stream { server { listen 12345; proxy_pass example.com:12345; proxy_protocol on; } } ``` ## Example ```nginx http { log_format combined '$proxy_protocol_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; #... server { server_name localhost; listen 80 proxy_protocol; listen 443 ssl proxy_protocol; ssl_certificate /etc/nginx/ssl/public.example.com.pem; ssl_certificate_key /etc/nginx/ssl/public.example.com.key; location /app/ { proxy_pass http://backend1; proxy_set_header Host $host; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-Forwarded-For $proxy_protocol_addr; } } } stream { log_format basic '$proxy_protocol_addr - $remote_user [$time_local] ' '$protocol $status $bytes_sent $bytes_received ' '$session_time'; #... server { listen 12345 ssl proxy_protocol; ssl_certificate /etc/nginx/ssl/cert.pem; ssl_certificate_key /etc/nginx/ssl/cert.key; proxy_pass backend.example.com:12345; proxy_protocol on; } } ``` The example assumes that there is a load balancer in front of NGINX to handle all incoming HTTPS traffic, for example Amazon ELB. NGINX accepts HTTPS traffic on port 443 (`listen 443 ssl;`), TCP traffic on port 12345, and accepts the client’s IP address passed from the load balancer via the PROXY protocol as well (the `proxy_protocol` parameter to the `listen` directive in both the `http {}` and `stream {}` blocks. NGINX terminates HTTPS traffic (the [`ssl_certificate`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate) and [`ssl_certificate_key`](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key) directives) and proxies the decrypted data to a backend server: - For HTTP: `proxy_pass http://backend1;` - For TCP: `proxy_pass backend.example.com:12345` It includes the client IP address and port with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directives. The [`$proxy_protocol_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_proxy_protocol_addr) variable specified in the [`log_format`](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) directive also passes the client’s IP address to the log for both HTTP and TCP. Additionally, a TCP server (the `stream {}` block) sends its own PROXY protocol data to its backend servers (the `proxy_protocol on` directive). -------------------------------------------------------------------------------- title: "NGINX Variables Index" url: https://docs.nginx.com/nginx/variables/index.md -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Restricting Access by Geographical Location" url: https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-by-geoip/index.md description: Control access or forward traffic to different upstream servers based on the client's geographical location, using the GeoIP2 dynamic module. -------------------------------------------------------------------------------- ## Introduction F5 NGINX Plus can differentiate users based on their geographical location. For example, you can have different website content for different countries, or you can restrict content distribution to a particular country or city. NGINX Plus uses third-party MaxMind databases to match the IP address of the user and its location. As soon as the geoposition is known, it is then possible to use geoip-based variables in the [map](https://nginx.org/en/docs/http/ngx_http_map_module.html) or the [split_clients](https://nginx.org/en/docs/http/ngx_http_split_clients_module.html) module. > **Note** MaxMind GeoLite Legacy databases are currently [discontinued](https://blog.maxmind.com/2018/01/discontinuation-of-the-geolite-legacy-databases), MaxMind GeoIP2 or GeoLite2 databases and NGINX Plus [GeoIP2 module](nginx/admin-guide/dynamic-modules/geoip2.md) should be used instead. Restricting by geographical location works both for HTTP and TCP/UDP protocols. ## Prerequisites - NGINX Plus [GeoIP2 dynamic module](nginx/admin-guide/dynamic-modules/geoip2.md) - [GeoIP2](https://www.maxmind.com/en/geoip2-databases) or [GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/) databases from MaxMind - (optional) [mmdblookup](http://maxmind.github.io/libmaxminddb/mmdblookup.html) utility that looks up an IP address in a MaxMind Database file ## Getting the Databases The GeoIP2 or GeoLite2 databases can be obtained from the [MaxMind download page](https://www.maxmind.com/en/geoip2-databases). In this example, the GeoLite2 free downloadable databases are used. To get and unpack GeoLite2 Country database: ```shell wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.mmdb.gz gunzip GeoLite2-Country.mmdb.gz ``` To get and unpack GeoLite2 City database: ```shell wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz gunzip GeoLite2-City.mmdb.gz ``` ## Understanding Database Structure To see the available geodata, you can query the GeoLite2-Country and GeoLite2-City databases with the [mmdblookup](http://maxmind.github.io/libmaxminddb/mmdblookup.html) utility. The geodata is represented as the JSON tree. Install the [libmaxminddb](http://maxmind.github.io/libmaxminddb/index.html) database utility: - For Amazon Linux, CentOS, Oracle Linux, and RHEL: ```shell yum install libmaxminddb-devel ``` - For Debian and Ubuntu: ```shell apt-get install libmaxminddb-dev ``` - For SLES: ```shell zypper install libmaxminddb-devel ``` A query to the databases can be sent in the following format: ```none mmdblookup –file [FILE PATH] –ip [IP ADDRESS] [DATA PATH] ``` For example, to get all available geodata for the `8.8.8.8` IP address, send the following command: ```shell mmdblookup --file /usr/local/etc/geoip2/GeoLite2-Country.mmdb --ip 8.8.8.8 ``` The output will be: ```json { "continent": { "code": "NA" "geoname_id": 6255149 "names": { "de": "Nordamerika" "en": "North America" "es": "Norteamérica" "fr": "Amérique du Nord" "ja": "北アメリカ" "pt-BR": "América do Norte" "ru": "Северная Америка" "zh-CN": "北美洲" } } "country": { "geoname_id": 6252001 "iso_code": "US" "names": { "de": "USA" "en": "United States" "es": "Estados Unidos" "fr": "États-Unis" "ja": "アメリカ合衆国" "pt-BR": "Estados Unidos" "ru": "США" "zh-CN": "美国" } } "registered_country": { "geoname_id": 6252001 "iso_code": "US" "names": { "de": "USA" "en": "United States" "es": "Estados Unidos" "fr": "États-Unis" "ja": "アメリカ合衆国" "pt-BR": "Estados Unidos" "ru": "США" "zh-CN": "美国" } } } ``` To get particular geodata, for example, only the ISO code of a particular country, add the `country iso_code` parameters to the end of the command: ```shell mmdblookup --file /usr/local/etc/geoip2/GeoLite2-Country.mmdb --ip 8.8.8.8 country iso_code ``` These parameters are also used when creating variables in the GeoIP2 module for NGINX. ## Configuring GeoIP2 in NGINX Plus 1. Install the GeoIP2 dynamic module for NGINX Plus: For Amazon Linux, CentOS, Oracle Linux, and RHEL: ```shell yum install nginx-plus-module-geoip2 ``` For Debian and Ubuntu: ```shell apt-get install nginx-plus-module-geoip2 ``` For SLES: ```shell zypper install nginx-plus-module-geoip2 ``` 2. Enable the GeoIP2 dynamic module in the NGINX Plus configuration file with the [load_module](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive specified in the `main` configuration level: ```nginx load_module modules/ngx_http_geoip2_module.so; load_module modules/ngx_stream_geoip2_module.so; http { # ... } ``` 3. Add the paths to the country and city databases to the NGINX configuration with the `geoip2 {}` block for [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http), [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream), or both: ```nginx http { #... geoip2 GeoIP2/GeoLite2-Country.mmdb { #... } geoip2 GeoIP2/GeoLite2-City.mmdb { #... } } stream { #... geoip2 GeoIP2/GeoLite2-Country.mmdb { #... } geoip2 GeoIP2/GeoLite2-City.mmdb { #... } } ``` 4. Basing on the [GeoIP database structure](#mmdblookup), create custom variables that will keep the data from the GeoIP2 database and then later pass the data to the [map](https://nginx.org/en/docs/http/ngx_http_map_module.html) or [split_clients](https://nginx.org/en/docs/http/ngx_http_split_clients_module.html) directives (can be applied in both the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) and [`stream {}`](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) contexts): ```nginx geoip2 GeoIP2/GeoLite2-City.mmdb { $geoip2_data_city_name city names en; $geoip2_data_postal_code postal code; $geoip2_data_latitude location latitude; $geoip2_data_longitude location longitude; $geoip2_data_state_name subdivisions 0 names en; $geoip2_data_state_code subdivisions 0 iso_code; } geoip2 GeoIP2/GeoLite2-Country.mmdb { $geoip2_data_continent_code continent code; $geoip2_data_country_iso_code country iso_code; } #... ``` ## Scenario: Choosing the Nearest Server Using the geolocation data from the created variables, a client connection can be redirected to the closest server, thus reducing network latency and improving connection speed. This can be achieved by using the continent code from the GeoIP2 database in a variable and the [map](https://nginx.org/en/docs/http/ngx_http_map_module.html) module that will create another variable whose value will be the closest server basing on a continent location. Basing on this value, NGINX will pass the request to the corresponding upstream server group. 1. Make sure you have configured the servers or [upstream server groups](nginx/admin-guide/load-balancer/http-load-balancer.md) for each continent, for example, `eu` for Europe, `na` for North America, `all` for cases when the IP address cannot be matched against the GeoIP database: ```nginx upstream all { server all1.example.com:12345; server all2.example.com:12345; } upstream eu { server eu1.example.com:12345; server eu2.example.com:12345; } upstream na { server na1.example.com:12345; server na2.example.com:12345; } ``` 2. Add the `geoip2 {}` block with a variable of any name (for example, `$geoip2_data_continent_code`) that obtains the continent code of the GeoIP2 database: ```nginx geoip2 GeoIP2/GeoLite2-Country.mmdb { $geoip2_data_continent_code continent code; } #... ``` 3. Create the [map](https://nginx.org/en/docs/http/ngx_http_map_module.html) block that will create the `$nearest_server` variable: ```nginx #... map $geoip2_data_continent_code $nearest_server { default all; EU eu; NA na; AS as; AF af; } #... ``` 3. Create the [`server {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) block which will pass the requests to one of the upstream server groups according to the value passed in the `$nearest_server` variable: ```nginx server { listen 12346; proxy_pass http://$nearest_server; } ``` If the continent is Europe, then the value of the `$nearest_server` will be `eu`, and the connection will be passed to the `eu` upstream via the [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive: ```nginx #... server { listen 12346; proxy_pass http://$nearest_server; } upstream all { server all1.example.com:12345; server all2.example.com:12345; upstream eu { server eu1.example.com:12345; server eu2.example.com:12345; } upstream na { server na1.example.com:12345; server na2.example.com:12345; } #... ``` ### Example This example can be applied in both the [http](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) and [stream](https://nginx.org/en/docs/stream/ngx_stream_core_module.html#stream) contexts. ```nginx # can be either "http {}" or "stream {}" #... geoip2 GeoIP2/GeoLite2-Country.mmdb { $geoip2_data_continent_code continent code; } map $geoip2_data_continent_code $nearest_server { default all; EU eu; NA na; AS as; AF af; } server { listen 12346; proxy_pass http://$nearest_server; } upstream all { server all1.example.com:12345; server all2.example.com:12345; } upstream eu { server eu1.example.com:12345; server eu2.example.com:12345; } upstream na { server na1.example.com:12345; server na2.example.com:12345; } ``` In this example, the IP address will be checked in the `GeoLite2-Country.mmdb` database, the result will be written to the `$geoip2_data_continent_code` variable. NGINX Plus will match the value of the variable against values in the [map](https://nginx.org/en/docs/http/ngx_http_map_module.html#map) directive and write the result in the custom variable, in our example `$nearest_server`. Basing on the value of the `$nearest_server`, the [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive will choose a corresponding upstream server. ## More Info - [GeoIP2 Dynamic Module Installation Instructions](/nginx/admin-guide/dynamic-modules/geoip2.md) - [MaxMind GeoIP2 Databases](https://www.maxmind.com/en/geoip2-databases) - [MaxMind Geolite2 Free Downloadable Databases](https://dev.maxmind.com/geoip/geoip2/geolite2/) -------------------------------------------------------------------------------- title: "Single Sign-On with Ping Identity" url: https://docs.nginx.com/nginx/deployment-guides/single-sign-on/ping-identity/index.md description: Enable OpenID Connect-based single sign-on (SSO) for applications proxied by NGINX Plus, using Ping Identity as the identity provider (IdP). -------------------------------------------------------------------------------- This guide explains how to enable single sign-on (SSO) for applications being proxied by F5 NGINX Plus. The solution uses OpenID Connect as the authentication mechanism, with [Ping Identity](https://www.pingidentity.com/en.html) (PingFederate or PingOne) as the Identity Provider (IdP), and NGINX Plus as the Relying Party. **Note:** This guide applies to [NGINX Plus Release 36](nginx/releases.md#r36) and later. In earlier versions, NGINX Plus relied on an [njs-based solution](#legacy-njs-guide), which required NGINX JavaScript files, key-value stores, and advanced OpenID Connect logic. In the latest NGINX Plus version, the new [OpenID Connect module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) simplifies this process to just a few directives. ## Prerequisites - [PingFederate](https://docs.pingidentity.com/pingfederate/latest/pf_pf_landing_page.html) Enterprise Federation Server or [PingOne](https://docs.pingidentity.com/pingone/p1_cloud__platform_main_landing_page.html) Cloud deployment with a Ping Identity account. - An NGINX Plus [subscription](https://www.f5.com/products/nginx/nginx-plus) and NGINX Plus [Release 36](nginx/releases.md#r36) or later. For installation instructions, see [Installing NGINX Plus](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/). - A domain name pointing to your NGINX Plus instance, for example, `demo.example.com`. ## Configure PingFederate or PingOne for Enterprise {#ping-create} **Note:** These steps outline an example with the cloud offering of **PingOne**. If you are using the on‑premises PingFederate, the user interface might slightly differ. Create a new application for NGINX Plus: 1. Log in to your Ping Identity admin console. 2. Go to **Applications** > **Applications**. 3. Click the **+** (plus) symbol to create a new OIDC Application. 4. On the New Application screen: - Enter the Name of your application, for example, `nginx-demo-app`. - Select the Application Type **OIDC Web App**. - Select **Save**. 5. In your OIDC application, Select the **Overview** tab: - in the **General** section, copy your **Client ID** and **Client Secret** values. You will need then later when configuring NGINX Plus. - In the **Connection Details** section, copy your **Issuer ID**. You will need it later when configuring NGINX Plus. For PingOne Cloud, the Issuer ID generally structured as `https://auth.pingone.com//as`. For PingFederate, the Issuer ID generally structured as `https://pingfederate.example.com:9031` appended with the realm path of your environment. 6. On the **Configuration** tab of your OIDC application: - In the **Redirect URIs** field, add the NGINX Plus callback URI, for example: `https://demo.example.com/oidc_callback`. - In the **Post Logout Redirect URIs** field, add the post logout redirect URI, for example: `https://demo.example.com/post_logout/`. - Select **Save**. 7. Assign the application to the appropriate **Groups** or **Users** who will be allowed to log in. ### Enable Front-Channel Logout (optional) {#ping-frontchannel-logout} Front-channel logout allows Ping Identity to notify NGINX Plus when a user signs out of other Single Logout (SLO)-participating applications or from their Ping portal. Ping sends a front-channel HTTP request (typically loaded in a hidden iframe) to a logout URL that you configure in the application settings. This feature is optional and requires NGINX Plus Release 36 or later, together with the `frontchannel_logout_uri` directive in the NGINX OIDC provider configuration. 1. In the Ping Identity admin console, open the OIDC application that you created earlier (for example, `nginx-demo-app`). 2. Locate the **Front-channel logout** or **Single Logout** settings for the application. In PingOne, these are available under the **Configuration** or **Policies** sections; in PingFederate they are available in the OAuth / OIDC application connection settings. The exact location and labels can vary between product versions. 3. Configure the front-channel logout URL to point to the NGINX Plus front-channel logout endpoint, for example: ```text https://demo.example.com/front_logout ``` 4. Ensure that the front-channel logout request includes both the issuer (`iss`) and the session identifier (`sid`) as query parameters, in accordance with the OpenID Connect front-channel logout specification. If your Ping Identity admin console exposes an option to **include session details** or **send session ID and issuer** in logout requests, enable it. 5. Save the application configuration. ### Get the OpenID Connect Discovery URL Check the OpenID Connect Discovery URL. By default, Ping Identity publishes the `.well-known/openid-configuration` document at the following address: For PingOne: `https://auth.pingone.com//as/.well-known/openid-configuration` For PingFederate: `https://pingfederate.example.com:9031//.well-known/openid-configuration` 1. Run the following `curl` command in a terminal: ```shell curl https://auth.pingone.com//as/.well-known/openid-configuration | jq ``` Where: - the `auth.pingone.com` is your PingOne server address (or your PingFederate server for on-premises) - the `` is your PingOne environment ID - the `/as` is the authorization server path - the `/.well-known/openid-configuration` is the default address for Ping Identity for document location - the `jq` command (optional) is used to format the JSON output for easier reading and requires the [jq](https://jqlang.github.io/jq/) JSON processor to be installed. The configuration metadata is returned in the JSON format: ```json { ... "issuer": "https://auth.pingone.com//as", "authorization_endpoint": "https://auth.pingone.com//as/authorize", "token_endpoint": "https://auth.pingone.com//as/token", "jwks_uri": "https://auth.pingone.com//as/jwks", "userinfo_endpoint": "https://auth.pingone.com//as/userinfo", "end_session_endpoint": "https://auth.pingone.com//as/signoff", ... } ``` 2. Copy the **issuer** value, you will need it later when configuring NGINX Plus. Typically, the OpenID Connect Issuer for PingOne is `https://auth.pingone.com//as`. **Note:** You will need the values of **Client ID**, **Client Secret**, and **Issuer** in the next steps. ## Set up NGINX Plus {#nginx-plus-setup} With PingOne or PingFederate configured, you can enable OIDC on NGINX Plus. NGINX Plus serves as the Rely Party (RP) application — a client service that verifies user identity. 1. Ensure that you are using the latest version of NGINX Plus by running the `nginx -v` command in a terminal: ```shell nginx -v ``` The output should match NGINX Plus Release 36 or later: ```none nginx version: nginx/1.29.3 (nginx-plus-r36) ``` 2. Ensure that you have the values of the **Client ID**, **Client Secret**, and **Issuer** obtained during [PingOne or PingFederate Configuration](#ping-create). 3. In your preferred text editor, open the NGINX configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). 4. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, make sure your public DNS resolver is specified with the [`resolver`](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive: By default, NGINX Plus re‑resolves DNS records at the frequency specified by time‑to‑live (TTL) in the record, but you can override the TTL value with the `valid` parameter: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; # ... } ``` 5. In the [`http {}`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context, define the PingOne or PingFederate provider named `ping` by specifying the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context: ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider ping { # ... } # ... } ``` 6. In the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context, specify: - your actual Ping **Client ID** obtained in [Ping Configuration](#ping-create) with the [`client_id`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id) directive - your **Client Secret** obtained in [Ping Configuration](#ping-create) with the [`client_secret`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive - the **Issuer** URL obtained in [Ping Configuration](#ping-create) with the [`issuer`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret) directive The `issuer` is typically your Ping Identity OIDC URL. For PingOne Cloud, the URL is `https://auth.pingone.com//as`. For PingFederate, the URL is `https://pingfederate.example.com:9031` followed by your environment's realm path. By default, NGINX Plus creates the OpenID metadata URL by appending the `/.well-known/openid-configuration` part to the Issuer URL. If your metadata URL is different, you can explicitly specify the metadata document with the [`config_url`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url) directive. - The **logout_uri** is URI that a user visits to start an RP‑initiated logout flow. - The **post_logout_uri** is absolute HTTPS URL where Ping Identity should redirect the user after a successful logout. This value **must also be configured** in the Ping Identity application's Post Logout Redirect URIs. - If the **logout_token_hint** directive set to `on`, NGINX Plus sends the user's ID token as a *hint* to Ping Identity. This directive is **required** by PingOne. - The **frontchannel_logout_uri** directive defines the URI that receives OpenID Connect front-channel logout requests from Ping Identity. This URI must be an HTTPS path and must match the front-channel logout URL configured for the application in Ping Identity. When Ping sends a front-channel logout GET request (typically via a hidden iframe) to this URI with the `iss` and `sid` query parameters, the OIDC module clears the corresponding user session on NGINX Plus. - If the **userinfo** directive is set to `on`, NGINX Plus will fetch userinfo from Ping Identity and append the claims from userinfo to the `$oidc_claims_` variables. - PKCE (Proof Key for Code Exchange) is automatically enabled when the provider's OpenID Connect discovery document advertises the `S256` code challenge method in the `code_challenge_methods_supported` field. You can override this behavior with the [`pkce`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce) directive: set `pkce off;` to disable PKCE even when `S256` is advertised, or `pkce on;` to force PKCE even if the IdP's metadata does not list `S256`. - The OIDC module automatically selects the client authentication method for the token endpoint based on the provider metadata `token_endpoint_auth_methods_supported`. When only `client_secret_post` is advertised, NGINX Plus uses the `client_secret_post` method and sends the client credentials in the POST body. When both `client_secret_basic` and `client_secret_post` are present, the module prefers HTTP Basic (`client_secret_basic`), which remains the default for Ping Identity. - **Note:** All interaction with the IdP is secured exclusively over SSL/TLS, so NGINX must trust the certificate presented by the IdP. By default, this trust is validated against your system’s CA bundle (the default CA store for your Linux or FreeBSD distribution). If the IdP’s certificate is not included in the system CA bundle, you can explicitly specify a trusted certificate or chain with the [`ssl_trusted_certificate`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate) directive so that NGINX can validate and trust the IdP’s certificate. ```nginx http { resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider ping { issuer https://auth.pingone.com//as; client_id ; client_secret ; logout_uri /logout; logout_token_hint on; post_logout_uri https://demo.example.com/post_logout/; frontchannel_logout_uri /front_logout; userinfo on; # Optional: PKCE configuration. By default, PKCE is automatically # enabled when the IdP advertises the S256 code challenge method. # pkce on; } # ... } ``` 7. Make sure you have configured a [server](https://nginx.org/en/docs/http/ngx_http_core_module.html#server) that corresponds to `demo.example.com`, and there is a [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) that [points](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) to your application (see [Step 10](#oidc_app)) at `http://127.0.0.1:8080` that is going to be OIDC-protected: ```nginx http { # ... server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # ... proxy_pass http://127.0.0.1:8080; } } # ... } ``` 8. Protect this [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) with Ping Identity OIDC by specifying the [`auth_oidc`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#auth_oidc) directive that will point to the `ping` configuration specified in the [`oidc_provider {}`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#oidc_provider) context in [Step 5](#ping-setup-oidc-provider): ```nginx # ... location / { auth_oidc ping; # ... proxy_pass http://127.0.0.1:8080; } # ... ``` 9. Pass the OIDC claims as headers to the application ([Step 10](#oidc_app)) with the [`proxy_set_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. These claims are extracted from the ID token returned by Ping: - [`$oidc_claim_sub`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - a unique `Subject` identifier assigned for each user by Ping Identity - [`$oidc_claim_email`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the e-mail address of the user - [`$oidc_claim_name`](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) - the full name of the user - any other OIDC claim using the [`$oidc_claim_ `](https://nginx.org/en/docs/http/ngx_http_oidc_module.html#var_oidc_claim_) variable **Note:** Ensure the `openid`, `profile`, `email` Scopes are enabled in Ping Identity. ```nginx # ... location / { auth_oidc ping; proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } # ... ``` 10. Provide endpoint for completing logout: ```nginx # ... location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } # ... ``` 11. Create a simple test application referenced by the `proxy_pass` directive which returns the authenticated user's full name and email upon successful authentication: ```nginx # ... server { listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nPing Identity sub: $http_sub\n"; default_type text/plain; } } ``` 12. Save the NGINX configuration file and reload the configuration: ```nginx nginx -s reload ``` ### Complete Example This configuration example summarizes the steps outlined above. It includes only essential settings such as specifying the DNS resolver, defining the OIDC provider, configuring SSL, and proxying requests to an internal server. ```nginx http { # Use a public DNS resolver for Issuer discovery, etc. resolver 10.0.0.1 ipv4=on valid=300s; oidc_provider ping { # The issuer is typically something like: # https://auth.pingone.com//as issuer https://auth.pingone.com//as; # Your Ping Identity Client ID and Secret client_id ; client_secret ; # RP‑initiated logout logout_uri /logout; post_logout_uri https://demo.example.com/post_logout/; logout_token_hint on; # Front-channel logout (OP‑initiated single sign-out) frontchannel_logout_uri /front_logout; # Fetch userinfo claims userinfo on; # Optional: PKCE configuration (enabled automatically when supported by the IdP) # pkce on; } server { listen 443 ssl; server_name demo.example.com; ssl_certificate /etc/ssl/certs/fullchain.pem; ssl_certificate_key /etc/ssl/private/key.pem; location / { # Enforce OIDC with Ping Identity auth_oidc ping; # Forward OIDC claims as headers if desired proxy_set_header sub $oidc_claim_sub; proxy_set_header email $oidc_claim_email; proxy_set_header name $oidc_claim_name; proxy_pass http://127.0.0.1:8080; } location /post_logout/ { return 200 "You have been logged out.\n"; default_type text/plain; } } server { # Simple test upstream server listen 8080; location / { return 200 "Hello, $http_name!\nEmail: $http_email\nPing Identity sub: $http_sub\n"; default_type text/plain; } } } ``` ### Testing 1. Open `https://demo.example.com/` in a browser. You will be automatically redirected to the PingOne sign-in page. 2. Enter valid Ping Identity credentials of a user who has access the application. Upon successful sign-in, PingOne redirects you back to NGINX Plus, and you will see the proxied application content (for example, "Hello, Jane Doe!"). 3. Navigate to `https://demo.example.com/logout`. NGINX Plus initiates an RP‑initiated logout; Ping Identity ends the session and redirects back to `https://demo.example.com/post_logout/`. 4. Refresh `https://demo.example.com/` again. You should be redirected to Ping Identity for a fresh sign‑in, proving the session has been terminated. ## Legacy njs-based Ping Identity Solution {#legacy-njs-guide} If you are running NGINX Plus R33 and earlier or if you still need the njs-based solution, refer to the [Legacy njs-based Ping Identity Guide](nginx/deployment-guides/single-sign-on/oidc-njs/ping-identity.md) for details. The solution uses the [`nginx-openid-connect`](https://github.com/nginxinc/nginx-openid-connect) GitHub repository and NGINX JavaScript files. ## See Also - [NGINX Plus Native OIDC Module Reference documentation](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [Release Notes for NGINX Plus R36](nginx/releases.md#r36) ## Revision History - Version 3 (November 2025) – Updated for NGINX Plus R36; added front-channel logout support (`frontchannel_logout_uri`), PKCE configuration (`pkce` directive), and the `client_secret_post` token endpoint authentication method. - Version 2 (August 2025) – Updated for NGINX Plus R35; added RP‑initiated logout (`logout_uri`, `post_logout_uri`, `logout_token_hint`) and `userinfo` support. - Version 1 (March 2025) – Initial version for NGINX Plus Release 34. -------------------------------------------------------------------------------- title: "Securing HTTP Traffic to Upstream Servers" url: https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/index.md description: Secure HTTP traffic between NGINX or F5 NGINX Plus and upstream servers, using SSL/TLS encryption. -------------------------------------------------------------------------------- This article explains how to encrypt HTTP traffic between NGINX and an upstream group or a proxied server. ## Prerequisites - [NGINX Open Source](https://nginx.org/en/download.html) or [F5 NGINX Plus](https://nginx.com/products/) - A [proxied server](nginx/admin-guide/web-server/reverse-proxy.md) or an [upstream group of servers](nginx/admin-guide/load-balancer/http-load-balancer.md) - SSL certificates and a private key ## Obtain SSL server certificates {#serever_certs} You can purchase a server certificate from a trusted certificate authority (CA). You can create own internal CA with an [OpenSSL](https://www.openssl.org/) library and generate your own certificate instead. The server certificate, along with a private key, should be placed on each upstream server. ## Obtain an SSL client certificate {#client_certs} NGINX will identify itself to the upstream servers with an SSL client certificate. A trusted CA must sign the client certificate, which is configured on NGINX together with the corresponding private key. You will also need to configure the upstream servers to require client certificates for all incoming SSL connections, and to trust the CA that issued NGINX client certificate. Then, when NGINX connects to the upstream, it will provide its client certificate and the upstream server will accept it. ## Configure NGINX {#config} First, change the URL to an upstream group to support SSL connections. In the NGINX configuration file, specify the `https` protocol in the [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive: ```nginx location /upstream { proxy_pass https://backend.example.com; } ``` Add the client certificate and the key that will be used to authenticate NGINX on each upstream server with [proxy_ssl_certificate](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_certificate) and [proxy_ssl_certificate_key](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_certificate_key) directives: ```nginx location /upstream { proxy_pass https://backend.example.com; proxy_ssl_certificate /etc/nginx/client.pem; proxy_ssl_certificate_key /etc/nginx/client.key; } ``` If you use a self-signed certificate for an upstream or your own CA, also include the [proxy_ssl_trusted_certificate](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_trusted_certificate). The file must be in the PEM format. Optionally, include the [proxy_ssl_verify](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_verify) and [proxy_ssl_verfiy_depth](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_verify_depth) directives to have NGINX check the validity of the security certificates: ```nginx location /upstream { #... proxy_ssl_trusted_certificate /etc/nginx/trusted_ca_cert.crt; proxy_ssl_verify on; proxy_ssl_verify_depth 2; #... } ``` Each new SSL connection requires a full SSL handshake between the client and server, which is quite CPU-intensive. To have NGINX proxy previously negotiated connection parameters and use a so-called abbreviated handshake, include the [proxy_ssl_session_reuse](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_session_reuse) directive: ```nginx location /upstream { #... proxy_ssl_session_reuse on; #... } ``` Optionally, you can specify which SSL protocols and ciphers are used: ```nginx location /upstream { #... proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; } ``` ## Configure upstream servers {#config_upstream} Each upstream server should be configured to accept HTTPS connections. For each upstream server, specify a path to the server certificate and the private key with [ssl_certificate](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate) and [ssl_certificate_key](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key) directives: ```nginx server { listen 443 ssl; server_name backend1.example.com; ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; #... location /yourapp { proxy_pass https://url_to_app.com; #... } } ``` Specify the path to a client certificate with the [ssl_client_certificate](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_client_certificate) directive: ```nginx server { #... ssl_client_certificate /etc/ssl/certs/ca.crt; ssl_verify_client optional; #... } ``` ## Complete example {#example} ```nginx http { #... upstream backend.example.com { server backend1.example.com:443; server backend2.example.com:443; } server { listen 80; server_name www.example.com; #... location /upstream { proxy_pass https://backend.example.com; proxy_ssl_certificate /etc/nginx/client.pem; proxy_ssl_certificate_key /etc/nginx/client.key; proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; proxy_ssl_trusted_certificate /etc/nginx/trusted_ca_cert.crt; proxy_ssl_verify on; proxy_ssl_verify_depth 2; proxy_ssl_session_reuse on; } } server { listen 443 ssl; server_name backend1.example.com; ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; ssl_client_certificate /etc/ssl/certs/ca.crt; ssl_verify_client optional; location /yourapp { proxy_pass https://url_to_app.com; #... } server { listen 443 ssl; server_name backend2.example.com; ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key; ssl_client_certificate /etc/ssl/certs/ca.crt; ssl_verify_client optional; location /yourapp { proxy_pass https://url_to_app.com; #... } } } ``` In this example, the “`https`” protocol in the [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive specifies that the traffic forwarded by NGINX to upstream servers be secured. When a secure connection is passed from NGINX to the upstream server for the first time, the full handshake process is performed: - The [proxy_ssl_certificate](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_certificate) directive defines the location of the PEM-format certificate required by the upstream server. - The [proxy_ssl_certificate_key](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_certificate_key) directive defines the location of the certificate’s private key. - The [proxy_ssl_protocols](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_protocols) and [proxy_ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_ciphers) directives control which protocols and ciphers are used. The next time NGINX passes a connection to the upstream server, session parameters will be reused because of the [proxy_ssl_session_reuse](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_session_reuse) directive, and the secured connection is established faster. The trusted CA certificates in the file named by the [proxy_ssl_trusted_certificate](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_trusted_certificate) directive are used to verify the certificate on the upstream. The [proxy_ssl_verify_depth](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_verify_depth) directive specifies that two certificates in the certificates chain are checked. The [proxy_ssl_verify](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_verify) directive verifies the validity of certificates. -------------------------------------------------------------------------------- title: "Securing TCP Traffic to Upstream Servers" url: https://docs.nginx.com/nginx/admin-guide/security-controls/securing-tcp-traffic-upstream/index.md description: Secure TCP traffic between NGINX or F5 F5 NGINX Plus and upstream servers, using SSL/TLS encryption. -------------------------------------------------------------------------------- This article explains how to secure TCP traffic between NGINX and an upstream TCP server or group of upstream TCP servers. ## Prerequisites - F5 NGINX Plus [R6](nginx/releases.md) or later, or the latest NGINX Open Source compiled with the `--with-stream` and `with-stream_ssl_module` configuration parameters - A proxied TCP server or an [upstream group of TCP servers](nginx/admin-guide/load-balancer/tcp-udp-load-balancer.md) - SSL certificates and a private key ## Obtaining SSL Server Certificates First, obtain SSL server certificates and a private key. Obtain an SSL server certificate from a trusted certificate authority (CA). Alternatively, generate one using an SSL library such as [OpenSSL](http://www.openssl.org/). Place the server certificates and private key on each of the upstream servers. Self-signed server certificates encrypt the connection between NGINX and the upstream server. However, these connections are vulnerable to a man-in-the-middle attack. (If an imposter impersonates the upstream server, NGINX will not know it is talking to a fake server.) To lessen the risk, obtain server certificates signed by a trusted CA. (You can create your own internal CA using OpenSSL.) Afterward, configure NGINX to only trust certificates signed by that CA. This makes it much more difficult for an attacker to impersonate an upstream server. ## Obtaining an SSL Client Certificate NGINX can identify itself to the upstream servers using an SSL client certificate. A trusted CA must sign this client certificate. It must be stored on NGINX along with the corresponding private key. Configure the upstream servers to require client certificates for all incoming SSL connections and trust the CA that issued the client certificate to NGINX. By doing so, NGINX provides its client certificate when it connects to an upstream server, and the server will accept the certificate. ## Configuring NGINX Open the NGINX configuration file. Then, include the [proxy_ssl](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl) directive in the `server` block on the `stream` level: ```nginx stream { server { ... proxy_pass backend; proxy_ssl on; } } ``` Specify the path to the SSL client certificate required by the upstream server. Also, specify the certificate’s private key: ```nginx server { ... proxy_ssl_certificate /etc/ssl/certs/backend.crt; proxy_ssl_certificate_key /etc/ssl/certs/backend.key; } ``` Optionally, specify which SSL protocols and ciphers to use: ```nginx server { ... proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; } ``` The [proxy_ssl_trusted_certificate](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_trusted_certificate) directive specifies the file containing the trusted CA certificates. These trusted certificates are used to validate the upstream server’s security certificates. The file must be in PEM format. Optionally, include the [proxy_ssl_verify](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_verify) and [proxy_ssl_verfiy_depth](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_verify_depth) directives so NGINX vaidates the security certificates: ```nginx server { ... proxy_ssl_trusted_certificate /etc/ssl/certs/trusted_ca_cert.crt; proxy_ssl_verify on; proxy_ssl_verify_depth 2; } ``` New SSL connections require a full SSL handshake between the client and server. This is quite CPU-intensive. NGINX can proxy previously negotiated connection parameters and use a so-called abbreviated handshake. This lessens the CPU load for new SSL connections. To enable this, include the [proxy_ssl_session_reuse](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_session_reuse) directive: ```nginx proxy_ssl_session_reuse on; ``` ## Complete Example ```nginx stream { upstream backend { server backend1.example.com:12345; server backend2.example.com:12345; server backend3.example.com:12345; } server { listen 12345; proxy_pass backend; proxy_ssl on; proxy_ssl_certificate /etc/ssl/certs/backend.crt; proxy_ssl_certificate_key /etc/ssl/certs/backend.key; proxy_ssl_protocols TLSv1.2 TLSv1.3; proxy_ssl_ciphers HIGH:!aNULL:!MD5; proxy_ssl_trusted_certificate /etc/ssl/certs/trusted_ca_cert.crt; proxy_ssl_verify on; proxy_ssl_verify_depth 2; proxy_ssl_session_reuse on; } } ``` In this example, the [proxy_ssl](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl) directive specifies to secure the TCP traffic NGINX forwards to upstream servers. A full handshake occurs when NGINX first secures a TCP connection to an upstream server. The upstream server asks NGINX to present a security certificate specified in the [proxy_ssl_certificate](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_certificate) directive. The [proxy_ssl_protocols](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_protocols) and [proxy_ssl_ciphers](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_ciphers) directives specify the respective protocols and ciphers. The [proxy_ssl_session_reuse](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_session_reuse) directive causes subsequent upstream connections to reuse the session parameters. This makes establishing a secured TCP connection faster. The [proxy_ssl_trusted_certificate](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_trusted_certificate) directive names a file containing trusted CA certificates. Use these certificates to verify the certificate on the upstream server. The [proxy_ssl_verify_depth](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_verify_depth) directive specifies to check two certificates in the certificates chain. The [proxy_ssl_verify](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_ssl_verify) directive verifies the validity of certificates. To learn more about NGINX Plus, please see our [commercial subscriptions](https://nginx.com/products/). -------------------------------------------------------------------------------- title: "Dynamic Denylisting of IP Addresses" url: https://docs.nginx.com/nginx/admin-guide/security-controls/denylisting-ip-addresses/index.md description: Control access to your site or apps from specific client IP addresses, using dynamic denylists built with the F5 NGINX Plus key-value store and API. -------------------------------------------------------------------------------- This section describes how to create a denylist or allowlist of specific client IP addresses, which denies or allows them access to your site, and how to dynamically maintain the list of addresses. ## Overview In F5 NGINX Plus [Release 13](nginx/releases.md#r13) (R13) and later, you can denylist some IP addresses as well as create and maintain a database of denylisted IP addresses. You can also explicitly allowlist other IP addresses. The IP addresses database is managed with the NGINX Plus API and keyval modules. NGINX Plus [Release 19](nginx/releases.md#r19) extends this capability by matching an IP address to any address within a subnet or network range. ## Prerequisites NGINX Plus [Release 13](nginx/releases.md#r13) and later, NGINX Plus [Release 19](nginx/releases.md#r19) and later for network ranges support. ## Setup First, enable the database for storing the list of denylisted and allowlisted IP addresses. 1. In NGINX Plus configuration file, include the [keyval_zone](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval_zone) directive in the [http](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context to create a memory zone for storing keys and values. This sample directive creates a 1‑MB zone called **one**. ```nginx http { # ... keyval_zone zone=one:1m; } ``` To perform matching of an IP address against subnets (for example, `192.168.13.0/24`), specify the `type=ip` parameter of the [keyval_zone](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval_zone) directive: ```nginx http { # ... keyval_zone zone=one:1m type=ip; } ``` Note that the size of [keyval_zone](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval_zone) should also be increased as the `type=ip` parameter also enables an extra index stored in the zone. You can optionally include the `state` parameter to create a file where the key‑value database is stored and so persists across NGINX Plus reloads and restarts; in this example, **one.keyval**: ```nginx keyval_zone zone=one:1m state=one.keyval; ``` 2. Enable the NGINX Plus API in read‑write mode with the [api](https://nginx.org/en/docs/http/ngx_http_api_module.html#api) directive: ```nginx # ... server { listen 80; server_name www.example.com; location /api { api write=on; } } ``` We strongly recommend [restricting access](/nginx/admin-guide/security-controls/controlling-access-proxied-http.md) to this location, for example by allowing access only from `localhost` (`127.0.0.1`), and by using HTTP basic authentication to restrict use of the `PATCH`, `POST`, and `DELETE` methods to a specified set of users: ```nginx # ... server { listen 80; server_name www.example.com; location /api { api write=on; allow 127.0.0.1; deny all; limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } } } ``` 3. Populate the key‑value database with the API's [POST](https://nginx.org/en/docs/http/ngx_http_api_module.html#postHttpKeyvalZoneData) method, supplying the data in JSON format. You can use the `curl` command as in the following example. If the zone is empty, you can enter several key‑value pairs at once; otherwise, pairs must be added one at a time. ```shell $ curl -X POST -d '{ "10.0.0.1": "1", "10.0.0.2": "1", "10.0.0.3": "0", "10.0.0.4": "0" }' -s http://www.example.com/api/6/http/keyvals/one ``` If you have specified matching of IP addresses against network ranges (with the `type=ip` parameter of the [keyval_zone](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval_zone) directive), send the `POST` command with the network range specified in CIDR notation: ```shell $ curl -X POST -d '{ "192.168.13.0/24": "1" }' -s http://www.example.com/api/6/http/keyvals/one ``` 4. Define how client IP addresses are evaluated against the key‑value database, by including the [keyval](https://nginx.org/en/docs/http/ngx_http_keyval_module.html#keyval) directive in the [http](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) context. The directive takes advantage of the standard NGINX and NGINX Plus variable [`$remote_addr`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_remote_addr), which is set to the client IP address automatically for every request. As it processes each request, NGINX Plus: - Looks up the first parameter (here, `$remote_addr`, preset to the client's IP address) in the key‑value database specified by the `zone=` parameter (here, **one**). - If a key in the database exactly matches `$remote_addr`, sets the second parameter (here, `$target`) to the value corresponding to the key. In our example, the value is `1` for denylisted addresses or `0` for allowlisted addresses. ```nginx http { # ... keyval_zone zone=one:1m type=ip state=one.keyval; keyval $remote_addr $target zone=one; # Client address is the key, # $target is the value; } ``` 5. Create a rule with the [if](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if) directive that either allows or denies access depending on the client IP address. With this rule, access is allowed when `$target` is `0` and denied when it is `1`: ```nginx if ($target) { return 403; } ``` ## Managing the Key-Value Database You can use API methods to update a key‑value database dynamically, without requiring a reload of NGINX Plus. All of the following examples operate on the **one** zone, which is accessible at ****. - To get the list of all database entries for a zone: ```shell curl -X GET 'http://www.example.com/api/6/http/keyvals/one' ``` - To update the value for an existing entry (in this example to change the access status for IP address `10.0.0.4` from allowlisted to denylisted): ```shell curl -X PATCH -d '{"10.0.0.4": "1"}' -s 'http://www.example.com/api/6/http/keyvals/one' ``` - To add an entry to a populated zone: ```shell curl -X POST -d '{"10.0.0.5": "1"}' -s 'http://www.example.com/api/6/http/keyvals/one' ``` - To delete an entry: ```shell curl -X PATCH -d '{"10.0.0.4":null}' -s 'http://www.example.com/api/6/http/keyvals/one' ``` ## Full Example The full NGINX Plus configuration: ```nginx http { # ... keyval_zone zone=one:1m type=ip state=one.keyval; keyval $remote_addr $target zone=one; server { listen 80; server_name www.example.com; location /api { api write=on; allow 127.0.0.1; deny all; limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } } if ($target) { return 403; } } } ``` This configuration: - Creates a 1 MB keyval zone **one** that accepts network ranges and also creates the file **one.keyval** to make the database of key‑value pairs persists across reloads and restarts of NGINX Plus. - Enables the NGINX Plus API in write mode so that the zone can populated with IP addresses. - Enables lookup of the IP address `$remote_addr` in the key-value database as the key, and puts the value of the found key into the `$target` variable. - Enables a simple rule to check for the resulting value: if the value of `$target` is `1` (address is denylisted), return `403 (Forbidden)` to the client. The following `curl` command populates the empty keyval zone **one** with IP addresses that are denylisted (value is `1`) or allowlisted (value is `0`): ```shell curl -X POST -d '{ "10.0.0.1": "1", "192.168.13.0/24": "1", "10.0.0.3": "0", "10.0.0.4": "0" }' -s 'http://www.example.com/api/6/http/keyvals/one' ``` ## See Also - [Dynamic IP Denylisting with NGINX Plus and fail2ban](https://www.nginx.com/blog/dynamic-ip-denylisting-with-nginx-plus-and-fail2ban/) -------------------------------------------------------------------------------- title: "NGINX Control REST API reference" url: https://docs.nginx.com/nginx/admin-guide/basic-functionality/control-api-reference/index.md -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "NGINX Plus REST API reference" url: https://docs.nginx.com/nginx/admin-guide/monitoring/nginx-plus-api-reference/index.md -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "F5 DoS for NGINX" url: https://docs.nginx.com/nginx-app-protect-dos/index.md -------------------------------------------------------------------------------- ## About Achieve comprehensive protection against DoS and DDoS attacks for your apps and APIs with a multi-layered, adaptive, automated mitigation strategy for DevOps environments. Running natively on NGINX Plus and NGINX Ingress Controller, F5 DoS for NGINX is platform-agnostic and supports deployment options ranging from edge load balancers to individual pods in Kubernetes clusters. ## Featured content [//]: # "You can add a maximum of three cards: any extra will not display." [//]: # "One card will take full width page: two will take half width each. Three will stack like an inverse pyramid." [//]: # "Some examples of content could be the latest release note, the most common install path, and a popular new feature." ### [Deployment](/nginx-app-protect-dos/deployment-guide/learn-about-deployment/) Read how to install and upgrade F5 DoS for NGINX ### [Troubleshooting](/nginx-app-protect-dos/troubleshooting/how-to-troubleshoot/) Learn how to debug F5 DoS for NGINX ### [Releases](/nginx-app-protect-dos/releases/) Review changelogs for F5 DoS for NGINX -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.10" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.10/index.md description: Release notes for F5 DoS for NGINX 4.10, including support for NGINX Plus R37.1. -------------------------------------------------------------------------------- F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) attacks for your web applications. ## Release 4.10 August 26, 2026 ### New features - NGINX Plus R37.1 support ### Supported packages | Distribution name | Package file | |----------------------------|-------------------------------------------------------| | Alpine 3.21 | _app-protect-dos-37+4.10.0-r1.apk_ | | Alpine 3.22 | _app-protect-dos-37+4.10.0-r1.apk_ | | Amazon Linux 2023 | _app-protect-dos-37+4.10.0-1.amzn2023.ngx.x86_64.rpm_ | | RHEL 8 and Rocky Linux 8 | _app-protect-dos-37+4.10.0-1.el8.ngx.x86_64.rpm_ | | RHEL 9 and Rocky Linux 9 | _app-protect-dos-37+4.10.0-1.el9.ngx.x86_64.rpm_ | | RHEL 10 and Rocky Linux 10 | _app-protect-dos-37+4.10.0-1.el10.ngx.x86_64.rpm_ | | Debian 11 | _app-protect-dos_37+4.10.0-1\~bullseye_amd64.deb_ | | Debian 12 | _app-protect-dos_37+4.10.0-1\~bookworm_amd64.deb_ | | Debian 13 | _app-protect-dos_37+4.10.0-1\~trixie_amd64.deb_ | | Ubuntu 22.04 | _app-protect-dos_37+4.10.0-1\~jammy_amd64.deb_ | | Ubuntu 24.04 | _app-protect-dos_37+4.10.0-1\~noble_amd64.deb_ | | NGINX Plus | _NGINX Plus R37.1_ | ### Important notes - F5 DoS for NGINX 4.10 requires NGINX Plus **R37.1**, whereas 4.9 requires **R37.0**. Each `nginx-plus-module-appprotectdos` package depends on a virtual package named `nginx-plus-r` that only its own NGINX Plus release provides, so F5 DoS for NGINX and NGINX Plus must be upgraded together. - On Debian and Ubuntu, `apt` considers only the newest `nginx-plus` available and will not select an older one on its own. When installing a specific F5 DoS for NGINX version, pin `nginx-plus` as well, or the install fails with `Unable to correct problems, you have held broken packages`. See the [F5 DoS for NGINX Deployment Guide](/nap-dos/deployment-guide/learn-about-deployment.md) for the exact commands. - If you use the L4 accelerated mitigation feature, pin `app-protect-dos-ebpf-manager` to the same version as `app-protect-dos`. Left unpinned, it installs the most recent version, which may not match the rest of your installation. -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.9" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.9/index.md description: Release notes for F5 DoS for NGINX 4.9, including new platform support for Debian 13, RHEL 10, Rocky Linux 10, and NGINX Plus R37.0. -------------------------------------------------------------------------------- F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) attacks for your web applications. ## Release 4.9 May 13, 2026 ### New features - NGINX Plus R37.0 support - Debian 13 support - RHEL 10 and Rocky Linux 10 support - Bug fixes ### Supported packages | Distribution name | Package file | |----------------------------|------------------------------------------------------| | Alpine 3.21 | _app-protect-dos-37+4.9.6-r1.apk_ | | Alpine 3.22 | _app-protect-dos-37+4.9.6-r1.apk_ | | Amazon Linux 2023 | _app-protect-dos-37+4.9.6-1.amzn2023.ngx.x86_64.rpm_ | | RHEL 8 and Rocky Linux 8 | _app-protect-dos-37+4.9.6-1.el8.ngx.x86_64.rpm_ | | RHEL 9 and Rocky Linux 9 | _app-protect-dos-37+4.9.6-1.el9.ngx.x86_64.rpm_ | | RHEL 10 and Rocky Linux 10 | _app-protect-dos-37+4.9.6-1.el10.ngx.x86_64.rpm_ | | Debian 11 | _app-protect-dos_37+4.9.6-1\~bullseye_amd64.deb_ | | Debian 12 | _app-protect-dos_37+4.9.6-1\~bookworm_amd64.deb_ | | Debian 13 | _app-protect-dos_37+4.9.6-1\~trixie_amd64.deb_ | | Ubuntu 22.04 | _app-protect-dos_37+4.9.6-1\~jammy_amd64.deb_ | | Ubuntu 24.04 | _app-protect-dos_37+4.9.6-1\~noble_amd64.deb_ | | NGINX Plus | _NGINX Plus R37.0_ | -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.8" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.8/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.8 F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.8 December 1, 2025 ### New features - R36 support - Add support for Alpine 3.22 - Remove support for Alpine 3.19 - Bugs fixing ### Supported packages | Distribution name | Package file | |--------------------------|------------------------------------------------------| | Alpine 3.21 | _app-protect-dos-36+4.8.3-r1.apk_ | | Alpine 3.22 | _app-protect-dos-36+4.8.3-r1.apk_ | | Amazon Linux 2023 | _app-protect-dos-36+4.8.3-1.amzn2023.ngx.x86_64.rpm_ | | RHEL 8 and Rocky Linux 8 | _app-protect-dos-36+4.8.3-1.el8.ngx.x86_64.rpm_ | | RHEL 9 and Rocky Linux 9 | _app-protect-dos-36+4.8.3-1.el9.ngx.x86_64.rpm_ | | Debian 11 | _app-protect-dos_36+4.8.3-1\~bullseye_amd64.deb_ | | Debian 12 | _app-protect-dos_36+4.8.3-1\~bookworm_amd64.deb_ | | Ubuntu 22.04 | _app-protect-dos_36+4.8.3-1\~jammy_amd64.deb_ | | Ubuntu 24.04 | _app-protect-dos_36+4.8.3-1\~noble_amd64.deb_ | | NGINX Plus | _NGINX Plus R36_ | -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.7" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.7/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.7 F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.7 August 13, 2025 ### New features - R35 support - Add support for Alpine 3.21 - Add support for Rocky 9 - Remove support for Ubuntu 20.04 (EOL) - Bugs fixing ### Supported packages | Distribution name | Package file | |--------------------------|------------------------------------------------------| | Alpine 3.19 | _app-protect-dos-35+4.7.3-r1.apk_ | | Alpine 3.21 | _app-protect-dos-35+4.7.3-r1.apk_ | | Amazon Linux 2023 | _app-protect-dos-35+4.7.3-1.amzn2023.ngx.x86_64.rpm_ | | RHEL 8 and Rocky Linux 8 | _app-protect-dos-35+4.7.3-1.el8.ngx.x86_64.rpm_ | | RHEL 9 and Rocky Linux 9 | _app-protect-dos-35+4.7.3-1.el9.ngx.x86_64.rpm_ | | Debian 11 | _app-protect-dos_35+4.7.3-1\~bullseye_amd64.deb_ | | Debian 12 | _app-protect-dos_35+4.7.3-1\~bookworm_amd64.deb_ | | Ubuntu 22.04 | _app-protect-dos_35+4.7.3-1\~jammy_amd64.deb_ | | Ubuntu 24.04 | _app-protect-dos_35+4.7.3-1\~noble_amd64.deb_ | | NGINX Plus | _NGINX Plus R35_ | -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.6" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.6/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.6 F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. --- ## Release 4.6 April 1, 2025 --- ### New features - Support for NGINX Plus R34 - Add support for AmazonLinux 2023 - Remove support for Alpine 3.17 - Bug fixes --- ### Supported packages | Distribution name | Package file | |--------------------------|------------------------------------------------------| | Alpine 3.19 | _app-protect-dos-34+4.6.1-r1.apk_ | | Amazon Linux 2023 | _app-protect-dos-34+4.6.1-1.amzn2023.ngx.x86_64.rpm_ | | RHEL 8 and Rocky Linux 8 | _app-protect-dos-34+4.6.1-1.el8.ngx.x86_64.rpm_ | | RHEL 9 | _app-protect-dos-34+4.6.1-1.el9.ngx.x86_64.rpm_ | | Debian 11 | _app-protect-dos_34+4.6.1-1\~bullseye_amd64.deb_ | | Debian 12 | _app-protect-dos_34+4.6.1-1\~bookworm_amd64.deb_ | | Ubuntu 20.04 | _app-protect-dos_34+4.6.1-1\~focal_amd64.deb_ | | Ubuntu 22.04 | _app-protect-dos_34+4.6.1-1\~jammy_amd64.deb_ | | Ubuntu 24.04 | _app-protect-dos_34+4.6.1-1\~noble_amd64.deb_ | | NGINX Plus | _NGINX Plus R34_ | -------------------------------------------------------------------------------- title: "F5 DoS for NGINX Arbitrator 1.2.0" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-1.2.0/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX Arbitrator v1.2.0 ## Arbitrator Service Release 1.2.0 Mar 24, 2025 ### Enhanced Sync Mechanism in ADMD Based on Arbitrator Version In this release, ADMD now dynamically selects the appropriate synchronization method based on the Arbitrator's version: Arbitrator 1.2.0 and above – ADMD uses Batch Sync, allowing all VSs to be synchronized in a single request, improving efficiency and reducing overhead. Arbitrator versions below 1.2.0 – ADMD continues to use per-VS Sync, ensuring compatibility with earlier versions. This enhancement optimizes synchronization performance while maintaining backward compatibility. -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.5" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.5/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.5 F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. --- ## Release 4.5 Nov 19, 2024 F5 DoS for NGINX 4.5 adds support for NGINX Plus R33. --- ### New features - Support for NGINX Plus R33 - Add support for Alpine 3.19 - Add support for Ubuntu 24.04 - Remove support for CentOS 7 / RHEL 7 - *eBPF Manager - Privileged Process for Secure Command Handling* This feature introduces the eBPF Manager, a process that securely handles eBPF commands on behalf of other processes. By allowing certain processes to operate without elevated privileges, it enhances system security. As part of this enhancement, NGINX now runs under a non-root user account, which is required for deploying specific security solutions. --- ### Supported packages | Distribution name | Package file | |--------------------------|--------------------------------------------------| | Alpine 3.17 / 3.19 | _app-protect-dos-33+4.5.2-r1.apk_ | | RHEL 8 and Rocky Linux 8 | _app-protect-dos-33+4.5.2-1.el8.ngx.x86_64.rpm_ | | RHEL 9 | _app-protect-dos-33+4.5.2-1.el9.ngx.x86_64.rpm_ | | Debian 11 | _app-protect-dos_33+4.5.2-1\~bullseye_amd64.deb_ | | Debian 12 | _app-protect-dos_33+4.5.2-1\~bookworm_amd64.deb_ | | Ubuntu 20.04 | _app-protect-dos_33+4.5.2-1\~focal_amd64.deb_ | | Ubuntu 22.04 | _app-protect-dos_33+4.5.2-1\~jammy_amd64.deb_ | | Ubuntu 24.04 | _app-protect-dos_33+4.5.2-1\~noble_amd64.deb_ | | NGINX Plus | _NGINX Plus R33_ | -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.4" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.4/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.4. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.4 May 29, 2024 In this release, F5 DoS for NGINX supports NGINX Plus R32. ### New Features - Support for NGINX Plus R32 - Support for Debian 12 - Support for RHEL 9 - [Allowlisted IPs](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#access-file-directive-app_protect_dos_access_file) - Support for up to 1000 protected objects, ensuring high scalability and efficient resource utilization. ### Supported Packages #### App Protect DoS ##### Alpine 3.17 - app-protect-dos-32+4.4.1-r1.apk ##### CentOS 7.4+ / RHEL 7.4+ - app-protect-dos-32+4.4.1-1.el7.ngx.x86_64.rpm ##### RHEL 8 and Rocky Linux 8 - app-protect-dos-32+4.4.1-1.el8.ngx.x86_64.rpm ##### RHEL 9 - app-protect-dos-32+4.4.1-1.el9.ngx.x86_64.rpm ##### Debian 11 - app-protect-dos_32+4.4.1-1~bullseye_amd64.deb ##### Debian 12 - app-protect-dos_32+4.4.1-1~bookworm_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_32+4.4.1-1~focal_amd64.deb ##### Ubuntu 22.04 - app-protect-dos_32+4.4.1-1~jammy_amd64.deb #### NGINX Plus - NGINX Plus R32 -------------------------------------------------------------------------------- title: "Deployment" url: https://docs.nginx.com/nginx-app-protect-dos/deployment-guide/learn-about-deployment/index.md description: Learn about F5 DoS for NGINX Deployment. -------------------------------------------------------------------------------- ## Overview F5 DoS for NGINX provides behavioral protection against DoS for your web applications.

This guide explains how to deploy F5 DoS for NGINX as well as upgrade App Protect DoS. ## Prerequisites F5 DoS for NGINX is available to the customers as a downloadable dynamic module at an additional cost. To purchase or add F5 DoS for NGINX to an existing NGINX Plus subscription, contact the NGINX sales team. NGINX Plus Release 24 and later supports F5 DoS for NGINX. F5 DoS for NGINX supports the following operating systems: - [RHEL 8.1+ / Rocky Linux 8](#rhel-8--rocky-linux-8-installation) - [RHEL 9.0+ / Rocky Linux 9](#rhel-9--rocky-linux-9-installation) - [Debian 11 (Bullseye)](#debian--ubuntu-installation) - [Debian 12 (Bookworm)](#debian--ubuntu-installation) - [Ubuntu 22.04 (Jammy)](#debian--ubuntu-installation) - [Ubuntu 24.04 (Noble)](#debian--ubuntu-installation) - [Alpine 3.21](#alpine-installation) - [Alpine 3.22](#alpine-installation) - [AmazonLinux 2023](#amazon-linux-2023-installation) The F5 DoS for NGINX package has the following dependencies: 1. **nginx-plus-module-appprotectdos** - NGINX Plus dynamic module for App Protect DoS 2. **libcurl** - Software library for HTTP access 3. **zeromq4** - Software library for fast, message-based applications 4. **boost** - The free peer-reviewed portable C++ source libraries 5. **openssl** - Toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocol 6. **libelf** - Software library for ELF access See the NGINX Plus full list of prerequisites for more details. F5 DoS for NGINX can be installed as a module to an existing NGINX Plus installation or as a complete NGINX Plus with App Protect DoS installation in a clean environment or to a system with F5 WAF for NGINX. **Note:** - gRPC, HTTP/2 and WebSocket protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for the attack to be detected. - Monitor directive `app_protect_dos_monitor` with proxy_protocol parameter can not be configured on Ubuntu 18.04. As a result, gRPC and HTTP/2 DoS protection for proxy_protocol configuration is not supported. - Regularly update the Operating System (OS) to avoid known OS vulnerabilities which may impact the service. ## Platform Security Considerations When deploying App Protect DoS on NGINX Plus take the following precautions to secure the platform. This avoids the risk of causing a Denial of Service condition or compromising the platform security. - Restrict permissions to the files on the F5 DoS for NGINX platform to user **nginx** and group **nginx**, especially for the sensitive areas containing the configuration. - Remove unnecessary remote access services on the platform. - Configure a Syslog destination on the same machine as App Protect DoS and proxy to an external destination. This avoids eavesdropping and [man-in-the-middle](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) attacks on the Syslog channel. ## Virtual machine or bare metal Deployment ### RHEL 8+ / Rocky Linux 8 Installation 1. If you already have NGINX packages in your system, back up your configs and logs: ```shell sudo cp -a /etc/nginx /etc/nginx-plus-backup sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` 1. Create the **/etc/ssl/nginx** directory: ```shell sudo mkdir -p /etc/ssl/nginx ``` 1. Create the **/etc/nginx/** directory for the JWT license file: ```shell sudo mkdir -p /etc/nginx/ ``` 1. Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. 1. Copy the downloaded **.crt** and **.key** files to the **/etc/ssl/nginx/** directory and make sure they are named **nginx-repo.crt** and **nginx-repo.key**: ```shell sudo cp .crt /etc/ssl/nginx/nginx-repo.crt sudo cp .key /etc/ssl/nginx/nginx-repo.key ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 5. Install prerequisite packages: ```shell sudo dnf install ca-certificates wget 6. Enable Yum repositories to pull F5 DoS for NGINX dependencies: For RHEL subscription: ```shell sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms sudo subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm ``` For RockyLinux: ```shell sudo dnf -y install epel-release ``` 8. Add NGINX Plus and NGINX App Protect DoS repository: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/nginx-plus-8.repo sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/app-protect-dos-8.repo ``` 9. In case of fresh installation, update the repository and install the most recent version of the NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo dnf install app-protect-dos ``` For L4 accelerated mitigation feature (RHEL 8.6+): ```shell sudo dnf install app-protect-dos-ebpf-manager ``` **Note:** L4 accelerated mitigation feature (RHEL 8.6+): - `app-protect-dos-ebpf-manager` run with root privileges. Alternatively, you can use the following command to list available versions: ```shell sudo dnf --showduplicates list app-protect-dos ``` Then, install a specific version from the output of command above. For example: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo dnf install app-protect-dos-37+4.10.0 # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo dnf install app-protect-dos-37+4.9.6 ``` If you use the L4 accelerated mitigation feature, pin `app-protect-dos-ebpf-manager` to the matching version. Left unpinned, it installs the most recent version, which may not match the rest of your installation: ```shell sudo dnf --showduplicates list app-protect-dos-ebpf-manager sudo dnf install app-protect-dos-ebpf-manager-37+4.9.6 ``` 10. In case of upgrading from previously installed NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo dnf remove nginx-plus sudo dnf install app-protect-dos sudo systemctl start nginx ``` **Note:** Make sure to restore configuration from `/etc/nginx-plus-backup` back to `/etc/nginx-plus`. 1. Check the `nginx` version to verify that NGINX Plus is installed correctly: ```shell nginx -v ``` 11. Check the App Protect DoS binary version to ensure that you have the right version installed correctly: ```shell sudo admd -v ``` 12. Load the F5 DoS for NGINX module on the main context in the `nginx.conf` file: ```nginx load_module modules/ngx_http_app_protect_dos_module.so; ``` 13. Enable F5 DoS for NGINX in an `http/server/location` context in the `nginx.conf` file: ```nginx app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_monitor uri=serv:80/; # Assuming server_name "serv" on port 80, with the root path "/" ``` 14. Enable the L4 accelerated mitigation feature (RHEL 8.6+) in an `http` context in the `nginx.conf` file: ```nginx app_protect_dos_accelerated_mitigation on; ``` 15. Configure the SELinux to allow App Protect DoS: a. Using the vi editor, create a file: ```shell vi app-protect-dos.te ``` b. Insert the following contents into the file that you have created: ```shell module app-protect-dos 2.0; require { type unconfined_t; type unconfined_service_t; type httpd_t; type tmpfs_t; type initrc_t; type initrc_state_t; class capability sys_resource; class shm { associate read unix_read unix_write write }; class file { read write }; } allow httpd_t initrc_state_t:file { read write }; allow httpd_t self:capability sys_resource; allow httpd_t tmpfs_t:file { read write }; allow httpd_t unconfined_service_t:shm { associate read unix_read unix_write write }; allow httpd_t unconfined_t:shm { associate read write unix_read unix_write }; allow httpd_t initrc_t:shm { associate read unix_read unix_write write }; ``` c. Run the following chain of commands: ```shell sudo checkmodule -M -m -o app-protect-dos.mod app-protect-dos.te && \ sudo semodule_package -o app-protect-dos.pp -m app-protect-dos.mod && \ sudo semodule -i app-protect-dos.pp; ``` For L4 accelerated mitigation feature: a. Using the vi editor, create a file: ```shell vi app-protect-dos-ebpf-manager.te ``` b. Insert the following contents into the file you have created: ```shell module app-protect-dos-ebpf-manager 1.0; require { type root_t; type httpd_t; type unconfined_service_t; class sock_file write; class unix_stream_socket connectto; class shm { unix_read unix_write }; } allow httpd_t root_t:sock_file write; allow httpd_t unconfined_service_t:shm { unix_read unix_write }; allow httpd_t unconfined_service_t:unix_stream_socket connectto; ``` c. Run the following chain of commands: ```shell sudo checkmodule -M -m -o app-protect-dos-ebpf-manager.mod app-protect-dos-ebpf-manager.te && \ sudo semodule_package -o app-protect-dos-ebpf-manager.pp -m app-protect-dos-ebpf-manager.mod && \ sudo semodule -i app-protect-dos-ebpf-manager.pp; ``` If you encounter any issues, refer to the [Troubleshooting Guide](/nap-dos/troubleshooting/how-to-troubleshoot.md). **Note:** Additional SELinux configuration may be required to allow NGINX Plus to listen on specific network ports, connect to upstreams, and send syslog entries to remote systems. Refer to the practices outlined in the [Using NGINX and NGINX Plus with SELinux](https://www.f5.com/company/blog/nginx/using-nginx-plus-with-selinux) article for details. 16. To enable the NGINX/App-Protect-DoS service to start at boot, run the command: ```shell sudo systemctl enable nginx.service ``` 17. Start the NGINX service: ```shell sudo systemctl start nginx ``` 18. L4 mitigation To enable the `app-protect-dos-ebpf-manager` service to start at boot, run the command: ```shell sudo systemctl enable nginx.service ``` Start the `app-protect-dos-ebpf-manager` service: ``` sudo systemctl start app-protect-dos-ebpf-manager ``` ### RHEL 9+ / Rocky Linux 9 Installation 1. If you already have NGINX packages on your system, back up your configs and logs: ```shell sudo cp -a /etc/nginx /etc/nginx-plus-backup sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` 1. Create the **/etc/ssl/nginx** directory: ```shell sudo mkdir -p /etc/ssl/nginx ``` 1. Create the **/etc/nginx/** directory for the JWT license file: ```shell sudo mkdir -p /etc/nginx/ ``` 1. Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. 1. Copy the downloaded **.crt** and **.key** files to the **/etc/ssl/nginx/** directory and make sure they are named **nginx-repo.crt** and **nginx-repo.key**: ```shell sudo cp .crt /etc/ssl/nginx/nginx-repo.crt sudo cp .key /etc/ssl/nginx/nginx-repo.key ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 5. Install prerequisite packages: ```shell sudo dnf install ca-certificates wget ``` 6. Enable the yum repositories to pull F5 DoS for NGINX dependencies: For RHEL subscription: ```shell sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms sudo subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm ``` For RockyLinux: ```shell sudo dnf -y install epel-release ``` 7. Add the NGINX Plus and NGINX App Protect DoS repositories: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/plus-9.repo sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/app-protect-dos-9.repo ``` 8. If you are performing a fresh installation, update the repository and install the most recent version of the NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo dnf install app-protect-dos ``` For L4 accelerated mitigation feature (RHEL 9): ```shell sudo dnf install app-protect-dos-ebpf-manager ``` **Note:** L4 accelerated mitigation feature (RHEL 9): - `app-protect-dos-ebpf-manager` run with root privileges. Alternatively, you can use the following command to list available versions: ```shell sudo dnf --showduplicates list app-protect-dos ``` Then, install a specific version from the output of command above. For example: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo dnf install app-protect-dos-37+4.10.0 # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo dnf install app-protect-dos-37+4.9.6 ``` If you use the L4 accelerated mitigation feature, pin `app-protect-dos-ebpf-manager` to the matching version. Left unpinned, it installs the most recent version, which may not match the rest of your installation: ```shell sudo dnf --showduplicates list app-protect-dos-ebpf-manager sudo dnf install app-protect-dos-ebpf-manager-37+4.9.6 ``` 9. In you are upgrading from previously installed NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo dnf remove nginx-plus sudo dnf install app-protect-dos sudo systemctl start nginx ``` **Note:** Make sure to restore configuration from `/etc/nginx-plus-backup` back to `/etc/nginx-plus`. 10. Check the NGINX binary version to ensure that you have NGINX Plus installed correctly: ```shell sudo nginx -v ``` 11. Check the App Protect DoS binary version to ensure that you have the right version installed correctly: ```shell sudo admd -v ``` 12. Load the F5 DoS for NGINX module on the main context in the `nginx.conf`: ```nginx load_module modules/ngx_http_app_protect_dos_module.so; ``` 13. Enable F5 DoS for NGINX on an `http/server/location` context in the `nginx.conf` file: ```nginx app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_monitor uri=serv:80/; # Assuming server_name "serv" on port 80, with the root path "/" ``` 14. Enable the L4 accelerated mitigation feature (RHEL 8.6+) in the `http` context of the `nginx.conf` file: ```nginx app_protect_dos_accelerated_mitigation on; ``` 15. Configure the SELinux to allow App Protect DoS: a. Using the vi editor, create a file: ```shell vi app-protect-dos.te ``` b. Insert the following contents into the file created above: ```shell module app-protect-dos 2.0; require { type unconfined_t; type unconfined_service_t; type httpd_t; type tmpfs_t; type initrc_t; type initrc_state_t; class capability sys_resource; class shm { associate read unix_read unix_write write }; class file { read write }; } allow httpd_t initrc_state_t:file { read write }; allow httpd_t self:capability sys_resource; allow httpd_t tmpfs_t:file { read write }; allow httpd_t unconfined_service_t:shm { associate read unix_read unix_write write }; allow httpd_t unconfined_t:shm { associate read write unix_read unix_write }; allow httpd_t initrc_t:shm { associate read unix_read unix_write write }; ``` c. Run the following chain of commands: ```shell sudo checkmodule -M -m -o app-protect-dos.mod app-protect-dos.te && \ sudo semodule_package -o app-protect-dos.pp -m app-protect-dos.mod && \ sudo semodule -i app-protect-dos.pp; ``` For L4 accelerated mitigation feature:
a. Using the vi editor, create a file: ```shell vi app-protect-dos-ebpf-manager.te ``` b. Insert the following contents into the file created above: ```shell module app-protect-dos-ebpf-manager 1.0; require { type root_t; type httpd_t; type unconfined_service_t; class sock_file write; class unix_stream_socket connectto; class shm { unix_read unix_write }; } allow httpd_t root_t:sock_file write; allow httpd_t unconfined_service_t:shm { unix_read unix_write }; allow httpd_t unconfined_service_t:unix_stream_socket connectto; ``` c. Run the following chain of commands: ```shell sudo checkmodule -M -m -o app-protect-dos-ebpf-manager.mod app-protect-dos-ebpf-manager.te && \ sudo semodule_package -o app-protect-dos-ebpf-manager.pp -m app-protect-dos-ebpf-manager.mod && \ sudo semodule -i app-protect-dos-ebpf-manager.pp; ``` If you encounter any issues, refer to the [Troubleshooting Guide](/nap-dos/troubleshooting/how-to-troubleshoot.md). **Note:** Additional SELinux configuration may be required to allow NGINX Plus to listen on specific network ports, connect to upstreams, and send syslog entries to remote systems. Refer to the practices outlined in the [Using NGINX and NGINX Plus with SELinux](https://www.f5.com/company/blog/nginx/using-nginx-plus-with-selinux/) article for details. 16. To enable the NGINX/App-Protect-DoS service to start at boot, run the command: ```shell sudo systemctl enable nginx.service ``` 17. Start the NGINX service: ```shell sudo systemctl start nginx ``` 18. L4 mitigation To enable the `app-protect-dos-ebpf-manager` service to start at boot, run the command: ```shell sudo systemctl enable nginx.service ``` Start the `app-protect-dos-ebpf-manager` service: ``` sudo systemctl start app-protect-dos-ebpf-manager ``` ### Debian / Ubuntu Installation 1. If you already have NGINX packages in your system, back up your configs and logs: ```shell sudo cp -a /etc/nginx /etc/nginx-plus-backup sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` 1. Create the **/etc/ssl/nginx** directory: ```shell sudo mkdir -p /etc/ssl/nginx ``` 1. Create the **/etc/nginx/** directory for the JWT license file: ```shell sudo mkdir -p /etc/nginx/ ``` 1. Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. 1. Copy the downloaded **.crt** and **.key** files to the **/etc/ssl/nginx/** directory and make sure they are named **nginx-repo.crt** and **nginx-repo.key**: ```shell sudo cp .crt /etc/ssl/nginx/nginx-repo.crt sudo cp .key /etc/ssl/nginx/nginx-repo.key ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 5. Install appropriate packages with `apt`: For Debian: ```shell sudo apt-get install apt-transport-https lsb-release ca-certificates wget gnupg2 debian-archive-keyring ``` For Ubuntu: ```shell sudo apt-get install apt-transport-https lsb-release ca-certificates wget gnupg2 ubuntu-keyring ``` **Note:** In case the apt installation or database update fails due to release info change, run the below command before you install. ```shell sudo apt-get update --allow-releaseinfo-change ``` 6. Download and add the NGINX signing key: ```shell sudo wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null ``` 7. Add NGINX Plus and F5 DoS for NGINX repository: For Debian: ```shell printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/debian `lsb_release -cs` nginx-plus\n" | sudo tee /etc/apt/sources.list.d/nginx-plus.list printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/debian `lsb_release -cs` nginx-plus\n" | sudo tee /etc/apt/sources.list.d/nginx-app-protect-dos.list ``` For Ubuntu: ```shell printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/ubuntu `lsb_release -cs` nginx-plus\n" | sudo tee /etc/apt/sources.list.d/nginx-plus.list printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/ubuntu `lsb_release -cs` nginx-plus\n" | sudo tee /etc/apt/sources.list.d/nginx-app-protect-dos.list ``` 8. Download the apt configuration to `/etc/apt/apt.conf.d`: ```shell sudo wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx ``` 9. In case of fresh Installation, update the repository and install the most recent version of the NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo apt-get update sudo apt-get install app-protect-dos ``` For L4 accelerated mitigation feature (Debian 11 / Debian 12 / Ubuntu 22.04 / Ubuntu 24.04): ```shell sudo apt-get install app-protect-dos-ebpf-manager ``` **Note:** L4 accelerated mitigation feature (Debian 11 / Debian 12 / Ubuntu 22.04 / Ubuntu 24.04): - `app-protect-dos-ebpf-manager` run with root privileges. Alternatively, to install a specific version, use the following commands to update and list the available versions: ```shell sudo apt-get update sudo apt-cache policy app-protect-dos ``` Each `nginx-plus-module-appprotectdos` package is built for one NGINX Plus release and depends on a virtual package named `nginx-plus-r`, which only that release of `nginx-plus` provides. To find which NGINX Plus release a version requires, list the dependencies of each available module version: ```shell sudo apt-cache show nginx-plus-module-appprotectdos | grep -E '^Version:|nginx-plus-r' ``` To find which `nginx-plus` versions provide that release, list its reverse provides. Replace `r37.0` with the release reported by the previous command: ```shell sudo apt-cache showpkg nginx-plus-r37.0 ``` Finally, install `app-protect-dos`, `nginx-plus-module-appprotectdos`, and `nginx-plus` in a single command, pinning all three. **Note:** `apt` considers only the newest `nginx-plus` available and does not select an older one to satisfy another package's dependency. If the version you are installing was built for an earlier NGINX Plus release, the install fails with `Unable to correct problems, you have held broken packages` unless you pin `nginx-plus` as well. The unpinned `apt-get install app-protect-dos` shown earlier needs no pin only while the most recent `app-protect-dos` is built for the most recent NGINX Plus release. If a new NGINX Plus release is published before the F5 DoS for NGINX version that supports it, pin both packages until that version is available. The two most recent versions and the NGINX Plus release each one requires: | F5 DoS for NGINX package | NGINX Plus release | `nginx-plus` version | |--------------------------|--------------------|----------------------| | 37+4.10.0 | R37.1 | 37.1.* | | 37+4.9.6 | R37.0 | 37.0.* | Quote any version that contains a `*` so that the shell does not expand it as a filename pattern. For example for Debian 11: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo apt-get install app-protect-dos=37+4.10.0-1~bullseye nginx-plus-module-appprotectdos=37+4.10.0-1~bullseye 'nginx-plus=37.1.*-1~bullseye' # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo apt-get install app-protect-dos=37+4.9.6-1~bullseye nginx-plus-module-appprotectdos=37+4.9.6-1~bullseye 'nginx-plus=37.0.*-1~bullseye' ``` For example, for Debian 12: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo apt-get install app-protect-dos=37+4.10.0-1~bookworm nginx-plus-module-appprotectdos=37+4.10.0-1~bookworm 'nginx-plus=37.1.*-1~bookworm' # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo apt-get install app-protect-dos=37+4.9.6-1~bookworm nginx-plus-module-appprotectdos=37+4.9.6-1~bookworm 'nginx-plus=37.0.*-1~bookworm' ``` For example for Ubuntu 22.04: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo apt-get install app-protect-dos=37+4.10.0-1~jammy nginx-plus-module-appprotectdos=37+4.10.0-1~jammy 'nginx-plus=37.1.*-1~jammy' # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo apt-get install app-protect-dos=37+4.9.6-1~jammy nginx-plus-module-appprotectdos=37+4.9.6-1~jammy 'nginx-plus=37.0.*-1~jammy' ``` For example for Ubuntu 24.04: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo apt-get install app-protect-dos=37+4.10.0-1~noble nginx-plus-module-appprotectdos=37+4.10.0-1~noble 'nginx-plus=37.1.*-1~noble' # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo apt-get install app-protect-dos=37+4.9.6-1~noble nginx-plus-module-appprotectdos=37+4.9.6-1~noble 'nginx-plus=37.0.*-1~noble' ``` If you use the L4 accelerated mitigation feature, pin `app-protect-dos-ebpf-manager` to the matching version. Left unpinned, it installs the most recent version, which may not match the rest of your installation. The `lsb_release` call fills in the suite of the system you are running on: ```shell sudo apt-cache policy app-protect-dos-ebpf-manager sudo apt-get install app-protect-dos-ebpf-manager=37+4.9.6-1~`lsb_release -cs` ``` 10. In the case of upgrading from a previously installed NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo apt-get update sudo apt-get remove nginx-plus sudo apt-get install app-protect-dos sudo systemctl start nginx ``` 11. Check the NGINX binary version to ensure that you have NGINX Plus installed correctly: ```shell sudo nginx -v ``` 12. Check the App Protect DoS binary version to ensure that you have the right version installed correctly: ```shell sudo admd -v ``` 13. Load the F5 DoS for NGINX module on the main context in the `nginx.conf` file: ```nginx load_module modules/ngx_http_app_protect_dos_module.so; ``` 14. Enable F5 DoS for NGINX on an `http/server/location` context in the `nginx.conf` via: ```nginx app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_monitor uri=serv:80/; # Assuming server_name "serv" on port 80, with the root path "/" ``` 15. Enable the L4 accelerated mitigation feature (Debian 11 / Debian 12 / Ubuntu 22.04 / Ubuntu 24.04) on the `http` context of the `nginx.conf` file: ```nginx app_protect_dos_accelerated_mitigation on; ``` 16. Start the NGINX service: ```shell sudo systemctl start nginx ``` 17. Start the L4 service: ```shell sudo systemctl start app-protect-dos-ebpf-manager ``` ### Alpine Installation 1. If you already have NGINX packages in your system, back up your configs and logs: ```shell sudo cp -a /etc/nginx /etc/nginx-plus-backup sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` 1. Create the **/etc/ssl/nginx** directory: ```shell sudo mkdir -p /etc/ssl/nginx ``` 1. Create the **/etc/nginx/** directory for the JWT license file: ```shell sudo mkdir -p /etc/nginx/ ``` 1. Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. 3. Upload `nginx-repo.key` to `/etc/apk/cert.key` and `nginx-repo.crt` to `/etc/apk/cert.pem`. Make sure that files do not contain other certificates and keys, as Alpine Linux does not support mixing client certificates for different repositories. 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 4. Add the NGINX public signing key to the directory `/etc/apk/keys`: ```shell sudo wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub ``` 5. Remove any previously configured NGINX Plus repository: ```shell sed "/plus-pkgs.nginx.com/d" /etc/apk/repositories ``` 6. Add NGINX Plus repository to `/etc/apk/repositories` file: ```shell printf "https://pkgs.nginx.com/plus/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | sudo tee -a /etc/apk/repositories ``` 7. Add F5 DoS for NGINX repository to `/etc/apk/repositories` file: ```shell printf "https://pkgs.nginx.com/app-protect-dos/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | sudo tee -a /etc/apk/repositories ``` 8. It is recommended to remove all community-supported NGINX packages. Note that all NGINX modules will be removed as well. ```shell sudo apk del -r app-protect-dos sudo apk del -r nginx ``` 9. Update the repository and install the most recent version of the NGINX Plus and F5 DoS for NGINX: ```shell sudo apk update sudo apk add nginx-plus app-protect-dos ``` For L4 accelerated mitigation feature: ```shell sudo apk add app-protect-dos-ebpf-manager ``` **Note:** L4 accelerated mitigation feature: - `app-protect-dos-ebpf-manager` run with root privileges. Alternatively, to install a specific version, use the following commands to update and list available versions: ```shell sudo apk update sudo apk info app-protect-dos ``` Finally, install a specific version from the output of command above. For example: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo apk add nginx-plus app-protect-dos=37+4.10.0-r1 # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo apk add nginx-plus app-protect-dos=37+4.9.6-r1 ``` If you use the L4 accelerated mitigation feature, pin `app-protect-dos-ebpf-manager` to the matching version. Left unpinned, it installs the most recent version, which may not match the rest of your installation: ```shell sudo apk info app-protect-dos-ebpf-manager sudo apk add app-protect-dos-ebpf-manager=37+4.9.6-r1 ``` 10. In case of upgrading from previously installed NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo apk update sudo apk del -r app-protect-dos sudo apk del -r nginx-plus sudo apk add nginx-plus app-protect-dos rc-service nginx-app-protect-dos start ``` 11. Check the NGINX binary version to ensure that you have NGINX Plus installed correctly: ```shell sudo nginx -v ``` 12. Check the App Protect DoS binary version to ensure that you have the right version installed correctly: ```shell sudo admd -v ``` 13. Load the F5 DoS for NGINX module on the main context in the `nginx.conf` file: ```nginx load_module modules/ngx_http_app_protect_dos_module.so; ``` 14. Enable F5 DoS for NGINX on an `http/server/location` context in the `nginx.conf` via: ```nginx app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_monitor uri=serv:80/; # Assuming server_name "serv" on port 80, with the root path "/" ``` 15. Enable the L4 accelerated mitigation feature on the `http` context of the `nginx.conf` file: ```nginx app_protect_dos_accelerated_mitigation on; ``` 16. Start the NGINX service: ```shell rc-service nginx-app-protect-dos start ``` 17. Start the L4 service: ```shell rc-service app-protect-dos-ebpf-manager start ``` ### Amazon Linux 2023 Installation 1. If you already have NGINX packages in your system, back up your configs and logs: ```shell sudo cp -a /etc/nginx /etc/nginx-plus-backup sudo cp -a /var/log/nginx /var/log/nginx-plus-backup ``` 1. Create the **/etc/ssl/nginx** directory: ```shell sudo mkdir -p /etc/ssl/nginx ``` 1. Create the **/etc/nginx/** directory for the JWT license file: ```shell sudo mkdir -p /etc/nginx/ ``` 1. Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. 1. Copy the downloaded **.crt** and **.key** files to the **/etc/ssl/nginx/** directory and make sure they are named **nginx-repo.crt** and **nginx-repo.key**: ```shell sudo cp .crt /etc/ssl/nginx/nginx-repo.crt sudo cp .key /etc/ssl/nginx/nginx-repo.key ``` 1. Copy the downloaded JWT file to the **/etc/nginx/** directory and make sure it is named **license.jwt**: ```shell sudo cp .jwt /etc/nginx/license.jwt ``` 5. Install prerequisite packages: ```shell sudo dnf install ca-certificates wget 6. Add NGINX Plus and F5 DoS for NGINX repository: ```shell sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/plus-amazonlinux2023.repo sudo wget -P /etc/yum.repos.d https://cs.nginx.com/static/files/app-protect-dos-amazonlinux2023.repo ``` 7. In case of fresh installation, update the repository and install the most recent version of the NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo dnf install app-protect-dos ``` For L4 accelerated mitigation feature: ```shell sudo dnf install app-protect-dos-ebpf-manager ``` **Note:** L4 accelerated mitigation feature: - `app-protect-dos-ebpf-manager` run with root privileges. Alternatively, you can use the following command to list available versions: ```shell sudo dnf --showduplicates list app-protect-dos ``` Then, install a specific version from the output of command above. For example: ```shell # F5 DoS for NGINX 4.10.0, which requires NGINX Plus R37.1 sudo dnf install app-protect-dos-37+4.10.0 # F5 DoS for NGINX 4.9.6, which requires NGINX Plus R37.0 sudo dnf install app-protect-dos-37+4.9.6 ``` If you use the L4 accelerated mitigation feature, pin `app-protect-dos-ebpf-manager` to the matching version. Left unpinned, it installs the most recent version, which may not match the rest of your installation: ```shell sudo dnf --showduplicates list app-protect-dos-ebpf-manager sudo dnf install app-protect-dos-ebpf-manager-37+4.9.6 ``` 8. In case of upgrading from previously installed NGINX Plus App Protect DoS package (which includes NGINX Plus): ```shell sudo dnf remove nginx-plus sudo dnf install app-protect-dos sudo systemctl start nginx ``` **Note:** Make sure to restore configuration from `/etc/nginx-plus-backup` back to `/etc/nginx-plus`. 9. Confirm the NGINX binary version to make sure that you have NGINX Plus installed correctly: ```shell sudo nginx -v ``` 10. Check the App Protect DoS binary version to ensure that you have the right version installed correctly: ```shell sudo admd -v ``` 11. Load the F5 DoS for NGINX module on the main context in the `nginx.conf` file: ```nginx load_module modules/ngx_http_app_protect_dos_module.so; ``` 12. Enable F5 DoS for NGINX in an `http/server/location` context in the `nginx.conf` file: ```nginx app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_monitor uri=serv:80/; # Assuming server_name "serv" on port 80, with the root path "/" ``` 13. Enable the L4 accelerated mitigation feature in an `http` context in the `nginx.conf` file: ```nginx app_protect_dos_accelerated_mitigation on; ``` 14. To enable the NGINX/App-Protect-DoS service to start at boot, run the command: ```shell sudo systemctl enable nginx.service ``` 15. Start the NGINX service: ```shell sudo systemctl start nginx ``` 16. L4 mitigation To enable the `app-protect-dos-ebpf-manager` service to start at boot, run the command: ```shell sudo systemctl enable nginx.service ``` Start the `app-protect-dos-ebpf-manager` service: ``` sudo systemctl start app-protect-dos-ebpf-manager ``` ## Docker Deployment ### Docker Deployment Instructions You need root permissions to execute the following steps. 1. Create a Dockerfile (see examples below) which copies the following files into the docker image: - `nginx-repo.crt`: Certificate for NGINX repository access - `nginx-repo.key`: Private key for NGINX repository access - `license.jwt`: JWT license file for NGINX Plus license management - `nginx.conf`: User defined `nginx.conf` with `app-protect-dos` enabled - `entrypoint.sh`: Docker startup script which spins up all App Protect DoS processes, must have executable permissions - custom_log_format.json: Optional user-defined security log format file (if not used - remove its references from the nginx.conf and Dockerfile) 2. Log in to NGINX Plus Customer Portal and download your `nginx-repo.crt`, `nginx-repo.key` and `license.jwt` files. 3. Copy the files to the directory where the Dockerfile is located. 4. Add F5 DoS for NGINX to your `nginx.conf`. The configuration below is an example for an `http` and `grpc+tls` servers which has F5 DoS for NGINX enabled. Note that every F5 DoS for NGINX related directive starts with `app_protect_dos_`. `nginx.conf` ```nginx user nginx; worker_processes auto; error_log /var/log/nginx/error.log error; worker_rlimit_nofile 65535; working_directory /tmp/cores; load_module modules/ngx_http_app_protect_dos_module.so; # F5 DoS for NGINX module events { worker_connections 65535; } http { include /etc/nginx/mime.types; log_format log_napd ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, ' 'outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ' 'ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; app_protect_dos_security_log_enable on; # Enable F5 DoS for NGINX's security logger app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" /var/log/adm/logger.log; # Security logger outputs to a file # app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=1.2.3.4:5261; # Security logger outputs to a syslog destination # HTTP/1 server server { default_type application/octet-stream; listen 80 reuseport; server_name serv80; set $loggable '0'; access_log /var/log/nginx/access.log log_napd if=$loggable; # Access log with rate limiting and additional information # access_log syslog:server=1.1.1.1:5561 log_napd if=$loggable; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; # Policy configuration for F5 DoS for NGINX location / { app_protect_dos_enable on; # Enable F5 DoS for NGINX in this block app_protect_dos_name "App80"; # PO name app_protect_dos_monitor uri=http://serv80/; # Health monitoring proxy_pass http://1.2.3.4:80; } } # gRPC server with ssl server { default_type application/grpc; listen 443 http2 ssl reuseport; server_name serv_grpc; # TLS config ssl_certificate /etc/ssl/certs/grpc.example.com.crt; ssl_certificate_key /etc/ssl/private/grpc.example.com.key; ssl_session_cache shared:SSL:10m; ssl_session_timeout 5m; ssl_ciphers HIGH:!aNULL:!MD5; ssl_protocols TLSv1.2 TLSv1.3; set $loggable '0'; access_log /var/log/nginx/access.log log_napd if=$loggable; #access_log syslog:server=1.1.1.1:5561 log_napd if=$loggable; location / { app_protect_dos_enable on; app_protect_dos_name "AppGRPC"; app_protect_dos_monitor uri=https://serv_grpc:443/service/method protocol=grpc; # mandatory for gRPC grpc_pass grpc://1.2.3.4:1001; } } sendfile on; tcp_nopush on; keepalive_timeout 65; } ``` **Note:** Make sure to replace upstream and proxy pass directives in this example with relevant application backend settings. 5. In the same directory create an `entrypoint.sh` file with executable permissions, with the following content: ```shell #!/usr/bin/env bash USER=nginx LOGDIR=/var/log/adm # prepare environment mkdir -p /var/run/adm /tmp/cores ${LOGDIR} chmod 755 /var/run/adm /tmp/cores ${LOGDIR} chown ${USER}:${USER} /var/run/adm /tmp/cores ${LOGDIR} # run processes /bin/su -s /bin/bash -c "/usr/bin/adminstall > ${LOGDIR}/adminstall.log 2>&1" ${USER} /bin/su -s /bin/bash -c "/usr/bin/admd -d --log info > ${LOGDIR}/admd.log 2>&1 &" ${USER} /usr/sbin/nginx -g 'daemon off;' ``` 6. Create a Docker image: ```shell DOCKER_BUILDKIT=1 docker build --no-cache --platform linux/amd64 --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key --secret id=license-jwt,src=./license.jwt -t app-protect-dos . ``` The `--no-cache` option tells Docker to build the image from scratch and ensures the installation of the latest version of NGINX Plus and F5 DoS for NGINX. If the Dockerfile was previously used to build an image without the `--no-cache` option, the new image uses versions from the previously built image from the Docker cache. For RHEL8/9 with subctiption manager setup add build arguments: ```shell DOCKER_BUILDKIT=1 docker build --build-arg RHEL_ORG=... --build-arg RHEL_ACTIVATION_KEY=... --no-cache --platform linux/amd64 --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key --secret id=license-jwt,src=./license.jwt -t app-protect-dos . ``` 8. Verify that the `app-protect-dos` image was created successfully with the docker images command: ```shell docker images app-protect-dos ``` 9. Create a container based on this image, for example, `my-app-protect-dos` container: ```shell docker run --name my-app-protect-dos -p 80:80 -d app-protect-dos ``` 10. Verify that the `my-app-protect-dos` container is up and running with the `docker ps` command: ```shell docker ps ``` 11. L4 Accelerated Mitigation Deployment Options:
There are three different ways to deploy the L4 accelerated mitigation feature:
1. Deploy in a Dedicated Container.
Create a shared folder on the host: ```shell mkdir /shared ``` This folder will be used to share data between containers. Modify the `entrypoint.sh` to run the L4 mitigation: ```shell # run processes /usr/bin/ebpf_manager_dos ``` Create and run the L4 container: ```shell docker run --privileged --network host --mount type=bind,source=/sys/fs/bpf,target=/sys/fs/bpf -v /shared:/shared --name my-app-protect-dos-ebpf-manager -d app-protect-dos-ebpf-manager ``` Create and run the main `app-protect-dos` container: ```shell docker run --name my-app-protect-dos -v /shared:/shared -p 80:80 -d app-protect-dos ``` 2. Deploy Directly on the Host.
To run L4 mitigation directly on the host:
1. Install the L4 mitigation on the host, as described in the OS-specific instructions. 2. Run the app-protect-dos container: ```shell docker run --name my-app-protect-dos -v /shared:/shared -p 80:80 -d app-protect-dos ``` 3. Run L4 Mitigation Inside the Same Container as `app-protect-dos`.
To run both L4 mitigation and the main application within the same container:
1. Modify the `entrypoint.sh`: ```shell ... # run processes /usr/bin/ebpf_manager_dos & ... ``` 2. run the container: ```shell docker run --name my-app-protect-dos -p 80:80 -d app-protect-dos ``` **Note:** L4 accelerated mitigation feature: - `app-protect-dos-ebpf-manager` need to run with root privileges. ### Alpine Docker Deployment Example ```Dockerfile # syntax=docker/dockerfile:1 # For Alpine 3.22: FROM alpine:3.22 # Download and add the NGINX signing keys: RUN wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub # Add NGINX Plus/F5 DoS for NGINX repository: RUN printf "https://pkgs.nginx.com/plus/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories && \ printf "https://pkgs.nginx.com/app-protect-dos/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories # Update the repository and install the most recent version of the F5 DoS for NGINX package (which includes NGINX Plus): RUN --mount=type=secret,id=nginx-crt,dst=/etc/apk/cert.pem,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/apk/cert.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apk update && apk add app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### AmazonLinux 2023 Docker Deployment Example ```Dockerfile # For AmazonLinux 2023: FROM amazonlinux:2023 # Install prerequisite packages: RUN dnf -y install ca-certificates # Add NGINX Plus/F5 DoS for NGINX repository: RUN curl -o /etc/yum.repos.d/plus-amazonlinux2023.repo https://cs.nginx.com/static/files/plus-amazonlinux2023.repo && \ curl -o /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo https://cs.nginx.com/static/files/app-protect-dos-amazonlinux2023.repo # Install F5 DoS for NGINX: RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf install -y app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ rm /etc/yum.repos.d/plus-amazonlinux2023.repo && \ rm /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo && \ dnf clean all && \ rm -rf /var/cache/dnf # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Debian 11 (Bullseye) / Debian 12 (Bookworm) Docker Deployment Example ```Dockerfile # Where can be bullseye/bookworm FROM debian:bullseye # Setup repository keys RUN mkdir -p /etc/ssl/nginx/ /etc/nginx/ && \ apt-get update && \ apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 debian-archive-keyring && \ wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null && \ printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/debian $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list && \ printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/debian $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-app-protect-dos.list && \ wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ apt-get remove --purge --auto-remove -y && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx-plus.list /etc/apt/sources.list.d/nginx-app-protect-dos.list && \ rm -rf /etc/apt/apt.conf.d/90nginx /var/lib/apt/lists/* # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Ubuntu 22.04 (Jammy) / 24.04 (Noble) Docker Deployment Example ```Dockerfile # Where version can be: jammy/noble FROM ubuntu:noble # Setup repository keys RUN apt-get update && \ apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 ubuntu-keyring && \ wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null && \ printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/ubuntu $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list && \ printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/ubuntu $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-app-protect-dos.list && \ wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ apt-get remove --purge --auto-remove -y && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx-plus.list /etc/apt/sources.list.d/nginx-app-protect-dos.list && \ rm -rf /etc/apt/apt.conf.d/90nginx /var/lib/apt/lists/* # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### RHEL 8 Docker Deployment Example ```Dockerfile # For UBI 8 FROM registry.access.redhat.com/ubi8 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Setup repository keys RUN subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} && \ subscription-manager refresh && \ subscription-manager attach --auto || true && \ subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms && \ subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms && \ dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && \ dnf -y install ca-certificates && \ curl -o /etc/yum.repos.d/plus-8.repo https://cs.nginx.com/static/files/plus-8.repo && \ curl -o /etc/yum.repos.d/app-protect-dos-8.repo https://cs.nginx.com/static/files/app-protect-dos-8.repo # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf -y install app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ rm /etc/yum.repos.d/plus-8.repo && \ rm /etc/yum.repos.d/app-protect-dos-8.repo && \ dnf clean all && \ rm -rf /var/cache/yum # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Rocky Linux 9 Docker Deployment Example ```Dockerfile # syntax=docker/dockerfile:1 # For Rocky Linux 9: FROM rockylinux:9 # Install prerequisite packages: RUN dnf -y install ca-certificates epel-release 'dnf-command(config-manager)' # Add NGINX App-protect-DoS & NGINX Plus repo to Yum: RUN curl -o /etc/yum.repos.d/plus-9.repo https://cs.nginx.com/static/files/plus-9.repo && \ curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo && \ dnf config-manager --set-enabled crb && \ dnf clean all # Install F5 DoS for NGINX: RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf install -y app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ rm /etc/yum.repos.d/plus-9.repo && \ rm /etc/yum.repos.d/app-protect-dos-9.repo && \ dnf clean all && \ rm -rf /var/cache/dnf # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ## Docker Deployment with NGINX App Protect ### Docker Deployment Instructions You need root permissions to execute the following steps. 1. Create a Dockerfile (see examples below) which copies the following files into the docker image: - `nginx-repo.crt`: Certificate for NGINX repository access - `nginx-repo.key`: Private key for NGINX repository access - `license.jwt`: JWT license file for NGINX Plus license management - `nginx.conf`: User defined `nginx.conf` with `app-protect-dos` enabled - `entrypoint.sh`: Docker startup script which spins up all App Protect DoS processes, must have executable permissions - `custom_log_format.json`: Optional user-defined security log format file (if not used - remove its references from the nginx.conf and Dockerfile) 2. Log in to NGINX Plus Customer Portal and download your `nginx-repo.crt`, `nginx-repo.key` and `license.jwt` files. 3. Copy the files to the directory where the Dockerfile is located. 4. Optionally, create `custom_log_format.json` in the same directory, for example: ```json { "filter": { "request_type": "all" }, "content": { "format": "splunk", "max_request_size": "any", "max_message_size": "10k" } } ``` 5. In the same directory create the `nginx.conf` file with the following contents: ```nginx user nginx; worker_processes auto; error_log /var/log/nginx/error.log error; worker_rlimit_nofile 65535; working_directory /tmp/cores; load_module modules/ngx_http_app_protect_module.so; load_module modules/ngx_http_app_protect_dos_module.so; events { worker_connections 65535; } http { include /etc/nginx/mime.types; log_format log_napd ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, ' 'outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ' 'ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; app_protect_dos_security_log_enable on; app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" /var/log/adm/logger.log; #app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=1.2.3.4:5261; # HTTP/1 server server { default_type application/octet-stream; listen 80 reuseport; server_name serv80; proxy_http_version 1.1; app_protect_policy_file "/etc/app_protect/conf/NginxDefaultPolicy.json"; app_protect_security_log_enable on; app_protect_security_log "/etc/nginx/custom_log_format.json" syslog:server=127.0.0.1:514; set $loggable '0'; access_log /var/log/nginx/access.log log_napd if=$loggable; #access_log syslog:server=1.1.1.1:5561 log_napd if=$loggable; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; location / { app_protect_dos_enable on; app_protect_dos_name "App80"; app_protect_dos_monitor uri=http://serv80/; proxy_pass http://1.2.3.4:80; } } # gRPC server with ssl server { default_type application/grpc; listen 443 http2 ssl reuseport; server_name serv_grpc; # TLS config ssl_certificate /etc/ssl/certs/grpc.example.com.crt; ssl_certificate_key /etc/ssl/private/grpc.example.com.key; ssl_session_cache shared:SSL:10m; ssl_session_timeout 5m; ssl_ciphers HIGH:!aNULL:!MD5; ssl_protocols TLSv1.2 TLSv1.3; set $loggable '0'; access_log /var/log/nginx/access.log log_napd if=$loggable; #access_log syslog:server=1.1.1.1:5561 log_napd if=$loggable; location / { app_protect_dos_enable on; app_protect_dos_name "AppGRPC"; app_protect_dos_monitor uri=https://serv_grpc:443/service/method protocol=grpc; # mandatory for gRPC grpc_pass grpc://1.2.3.4:1001; } } sendfile on; tcp_nopush on; keepalive_timeout 65; } ``` **Note:** Make sure to replace upstream and proxy pass directives in this example with relevant application backend settings. 6. For the L4 accelerated mitigation feature:
The following line in the `nginx.conf` file needs to be modified:
Change: ```nginx user nginx; ``` To: ```nginx user root; ``` 7. In the same directory create an `entrypoint.sh` file with executable permissions, with the following content: For Alpine / Debian / Ubuntu / UBI 8 / UBI 9: ```shell #!/usr/bin/env bash USER=nginx LOGDIR=/var/log/adm # prepare environment mkdir -p /var/run/adm /tmp/cores ${LOGDIR} chmod 755 /var/run/adm /tmp/cores ${LOGDIR} chown ${USER}:${USER} /var/run/adm /tmp/cores ${LOGDIR} # run processes /bin/su -s /bin/bash -c "/usr/bin/adminstall > ${LOGDIR}/adminstall.log 2>&1" ${USER} /bin/su -s /bin/bash -c "/opt/app_protect/bin/bd_agent &" ${USER} /bin/su -s /bin/bash -c "/usr/share/ts/bin/bd-socket-plugin tmm_count 4 proc_cpuinfo_cpu_mhz 2000000 total_xml_memory 307200000 total_umu_max_size 3129344 sys_max_account_id 1024 no_static_config 2>&1 > /var/log/app_protect/bd-socket-plugin.log &" ${USER} /bin/su -s /bin/bash -c "/usr/bin/admd -d --log info > ${LOGDIR}/admd.log 2>&1 &" ${USER} /usr/sbin/nginx -g 'daemon off;' ``` 8. Create a Docker image: For Debian/Ubuntu/Alpine/Amazon Linux: ```shell DOCKER_BUILDKIT=1 docker build --no-cache --platform linux/amd64 --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key --secret id=license-jwt,src=./license.jwt -t app-protect-dos . ``` For RHEL: ```shell DOCKER_BUILDKIT=1 docker build --build-arg RHEL_ORG=... --build-arg RHEL_ACTIVATION_KEY=... --no-cache --platform linux/amd64 --secret id=nginx-crt,src=nginx-repo.crt --secret id=nginx-key,src=nginx-repo.key --secret id=license-jwt,src=./license.jwt -t app-protect-dos . ``` **Notes:** - The `--no-cache` option tells Docker/Podman to build the image from scratch and ensures the installation of the latest version of NGINX Plus and F5 WAF for NGINX 4.x. If the Dockerfile was previously used to build an image without the `--no-cache` option, the new image uses versions from the previously built image from the cache. - For RHEL:
The subscription-manager is disabled when running inside containers based on Red Hat Universal Base images. You will need a registered and subscribed RHEL system. 9. Verify that the `app-protect-dos` image was created successfully with the docker images command: ```shell docker images app-protect-dos ``` 10. Create a container based on this image, for example, `my-app-protect-dos` container: ```shell docker run --name my-app-protect-dos -p 80:80 -d app-protect-dos ``` 11. Verify that the `my-app-protect-dos` container is up and running with the `docker ps` command: ```shell docker ps ``` ### Alpine Dockerfile example ```dockerfile # syntax=docker/dockerfile:1 # For Alpine 3.22: FROM alpine:3.22 # Download and add the NGINX signing keys: RUN wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub && \ wget -O /etc/apk/keys/app-protect-security-updates.rsa.pub https://cs.nginx.com/static/keys/app-protect-security-updates.rsa.pub # Add NGINX Plus repository: RUN printf "https://pkgs.nginx.com/plus/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories # Add F5 WAF for NGINX & Dos repositories: RUN printf "https://pkgs.nginx.com/app-protect-dos/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories && \ printf "https://pkgs.nginx.com/app-protect/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories && \ printf "https://pkgs.nginx.com/app-protect-security-updates/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories # Update the repository and install the most recent versions of the F5 WAF and F5 DoS for NGINX packages (which include NGINX Plus): RUN --mount=type=secret,id=nginx-crt,dst=/etc/apk/cert.pem,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/apk/cert.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apk update && apk add app-protect app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Amazon Linux Dockerfile example ```dockerfile # syntax=docker/dockerfile:1 FROM amazonlinux:2023 # Install prerequisite packages: RUN dnf -y install ca-certificates # Add NGINX/NAP WAF/NAP DOS repositories: RUN curl -o /etc/yum.repos.d/plus-amazonlinux2023.repo https://cs.nginx.com/static/files/plus-amazonlinux2023.repo && \ curl -o /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo https://cs.nginx.com/static/files/app-protect-dos-amazonlinux2023.repo && \ curl -o /etc/yum.repos.d/app-protect-amazonlinux2023.repo https://cs.nginx.com/static/files/app-protect-amazonlinux2023.repo && \ curl -o /etc/yum.repos.d/dependencies.amazonlinux2023.repo https://cs.nginx.com/static/files/dependencies.amazonlinux2023.repo # Update the repository and install the most recent versions of the F5 WAF and F5 DoS for NGINX packages (which include NGINX Plus): RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf -y install app-protect app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ rm /etc/yum.repos.d/plus-amazonlinux2023.repo && \ rm /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo && \ dnf clean all && \ rm -rf /var/cache/dnf && \ rm -rf /var/cache/yum # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Debian Docker Deployment Example ```Dockerfile # Where version can be: bullseye/bookworm FROM debian:bullseye # Install prerequisite packages: RUN apt-get update && \ apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 debian-archive-keyring && \ wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null && \ wget -qO - https://cs.nginx.com/static/keys/app-protect-security-updates.key | gpg --dearmor | tee /usr/share/keyrings/app-protect-security-updates.gpg > /dev/null # Add NGINX Plus, NGINX App Protect and F5 DoS for NGINX repository: RUN printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/debian `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-plus.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/debian `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect/debian `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-app-protect.list \ && printf "deb [signed-by=/usr/share/keyrings/app-protect-security-updates.gpg] https://pkgs.nginx.com/app-protect-security-updates/debian `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/app-protect-security-updates.list # Download the apt configuration to `/etc/apt/apt.conf.d`: RUN wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx # Update the repository and install the most recent versions of the F5 WAF and F5 DoS for NGINX packages (which includes NGINX Plus): RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y app-protect app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ apt-get remove --purge --auto-remove -y && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx-plus.list /etc/apt/sources.list.d/nginx-app-protect.list /etc/apt/sources.list.d/nginx-app-protect-dos.list && \ rm -rf /etc/apt/apt.conf.d/90nginx /var/lib/apt/lists/* # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Ubuntu Docker Deployment Example ```Dockerfile # Where version can be:jammy/noble FROM ubuntu:noble # Install prerequisite packages: RUN apt-get update && \ apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 ubuntu-keyring && \ wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null && \ wget -qO - https://cs.nginx.com/static/keys/app-protect-security-updates.key | gpg --dearmor | tee /usr/share/keyrings/app-protect-security-updates.gpg > /dev/null # Add NGINX Plus, NGINX App Protect and F5 DoS for NGINX repository: RUN printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/ubuntu `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-plus.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/ubuntu `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect/ubuntu `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-app-protect.list \ && printf "deb [signed-by=/usr/share/keyrings/app-protect-security-updates.gpg] https://pkgs.nginx.com/app-protect-security-updates/debian `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/app-protect-security-updates.list # Download the apt configuration to `/etc/apt/apt.conf.d`: RUN wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx # Update the repository and install the most recent versions of the F5 WAF and F5 DoS for NGINX packages (which include NGINX Plus): RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y app-protect app-protect-dos && \ cat license.jwt > /etc/nginx/license.jwt && \ apt-get remove --purge --auto-remove -y && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx-plus.list /etc/apt/sources.list.d/nginx-app-protect.list /etc/apt/sources.list.d/nginx-app-protect-dos.list && \ rm -rf /etc/apt/apt.conf.d/90nginx /var/lib/apt/lists/* # Forward request logs to Docker log collector: RUN ln -sf /dev/stdout /var/log/nginx/access.log && \ ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ## F5 DoS for NGINX Arbitrator ### Overview F5 DoS for NGINX arbitrator orchestrates all the running F5 DoS for NGINX instances to synchronize local/global attack start/stop. F5 DoS for NGINX arbitrator serves as a central coordinating component for managing multiple instances of App Protect DoS in a network. It is needed when there are more than one F5 DoS for NGINX instances. Its primary function is to ensure that all instances are aware of and share the same state for each protected object. Here's a clearer breakdown of how it works and why it's necessary: How F5 DoS for NGINX Arbitrator Works: - **Collecting State Periodically**: The arbitrator regularly collects the state information from all running instances of App Protect DoS. This collection occurs at set intervals, typically every 10 seconds. - **State Initialization for New Instances**: When a new App Protect DoS instance is created, it doesn't start with a blank or uninitialized state for a protected object. Instead, it retrieves the initial state for the protected object from the arbitrator. - **Updating State in Case of an Attack**: If an attack is detected by one of the App Protect DoS instances, that instance sends an attack notification to the arbitrator. The arbitrator then updates the state of the affected protected object to indicate that it is under attack. Importantly, this updated state is propagated to all other instances. ### Why F5 DoS for NGINX Arbitrator is Necessary F5 DoS for NGINX Arbitrator is essential for several reasons: - **Global State Management**: Without the arbitrator, each individual instance of App Protect DoS would manage its own isolated state for each protected object. This isolation could lead to inconsistencies. For example, if instance A declared an attack on a protected object named "PO-Example," instance B would remain unaware of this attack, potentially leaving the object vulnerable. - **Uniform Attack Detection**: With the arbitrator in place, when instance A detects an attack on "PO-Example" and reports it to the arbitrator, the state of "PO-Example" is immediately updated to indicate an attack. This means that all instances, including instance B, are aware of the attack and can take appropriate measures to mitigate it. In summary, F5 DoS for NGINX Arbitrator acts as a central coordinator to maintain a consistent and up-to-date global state for protected objects across multiple instances of App Protect DoS. This coordination helps ensure that attacks are properly detected and mitigated, and that knowledge gained by one instance is efficiently shared with others, enhancing the overall security of the network. ### F5 DoS for NGINX Arbitrator Deployment 1. Pull the official F5 DoS for NGINX Arbitrator image with the command: ```shell docker pull docker-registry.nginx.com/nap-dos/app_protect_dos_arb:latest ``` 2. Create a container based on this image, for example, `app-protect-dos-arb` container: ```shell docker run --name app_protect_dos_arb -p 3000:3000 -d docker-registry.nginx.com/nap-dos/app_protect_dos_arb ``` 3. Verify that the `app-protect-dos-arb` container is up and running with the `docker ps` command. 4. DNS records are required for F5 DoS for NGINX Arbitrator to work properly and be accessible by F5 DoS for NGINX servers. Ensure that the `svc-appprotect-dos-arb` or configured Arbitrator FQDN (with `app_protect_dos_arb_fqdn` directive) has a valid DNS resolution. This step is necessary only for VM/Docker deployments with arbitrator. When the arbitrator is in the same Kubernetes namespace as F5 DoS for NGINX, this step is not needed. ### Multi-VM Deployment The Arbitrator service is standalone. Once it is down, it can be seamlessly re-started. It will immediately recover all the needed information from F5 DoS for NGINX instances that communicate to it every 10 sec. It’s downtime is around 10-20 seconds which will not affect the F5 DoS for NGINX working. F5 DoS for NGINX Arbitrator service connects to port 3000 and can be seen under App Protect DoS instances. All modules try to connect to this service automatically. If it’s not accessible, each instance works in standalone mode. There is no such option for authentications between F5 DoS for NGINX servers and Arbitrator service like MTLS or password . Currently Arbitrator service is not exposed outside of the namespace. It is customers responsibility to isolate it from outside. It is applicable to any deployment of Arbitrator, not only to multi-VM. ## Post-Installation Checks You can run the following commands to ensure that F5 DoS for NGINX enforcement is operational. 1. Check that the three processes needed for F5 DoS for NGINX are running using `ps aux`: - admd - nginx: master process - nginx: worker process ```shell USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND nginx 7759 0.0 0.0 113120 1200 ? Ss Sep06 0:00 /bin/sh -c /usr/bin/admd -d --log info > /var/log/adm/admd.log 2>&1 root 7765 0.0 0.0 87964 1464 ? Ss Sep06 0:00 nginx: master process /usr/sbin/nginx -g daemon off; nginx 7767 0.0 0.1 615868 8188 ? Sl Sep06 0:04 nginx: worker process ``` 2. Verify that there are no NGINX errors in the `/var/log/nginx/error.log` and that the policy compiled successfully: ```shell 2020/09/07 15:33:44 [notice] 9307#9307: using the "epoll" event method 2020/09/07 15:33:44 [notice] 9307#9307: nginx/1.19.0 (nginx-plus-r22) 2020/09/07 15:33:44 [notice] 9307#9307: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 2020/09/07 15:33:44 [notice] 9307#9307: OS: Linux 3.10.0-327.28.3.el7.x86_64 2020/09/07 15:33:44 [notice] 9307#9307: getrlimit(RLIMIT_NOFILE): 1024:4096 2020/09/07 15:33:44 [notice] 9310#9310: start worker processes 2020/09/07 15:33:44 [notice] 9310#9310: start worker process 9311 PID <9311>, WORKER <0>, Function adm_ngx_init_process, line 684, version: 22+1.19.4-1.el7.ngx ``` 3. Check that by applying an attack, the attacker IP addresses are blocked while the good traffic pass through: a. Simulate good traffic: ```shell echo "Start Good Traffic 2" while true; do curl ${VS}/good1 & curl ${VS}/good2 & curl ${VS}/good3 & curl ${VS}/good4 sleep 0.1 done & ``` b. After 7 minutes start the attack: ```shell while [ true ] do ab -B ${BAD_IP1} -l -r -n 1000000 -c 150 -d -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ & ab -B ${BAD_IP2} -l -r -n 1000000 -c 150 -d -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ & ab -B ${BAD_IP3} -l -r -n 1000000 -c 150 -d -s 10 -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ killall ab done ``` c. See that the good traffic continue as usual while the attackers receive denial of service. To check F5 WAF for NGINX alongside F5 DoS for NGINX, just perform the normal tests as specified at [Admin Guide](https://docs.nginx.com/waf/install/virtual-environment/#post-installation-checks) ### Compatibility with NGINX Plus Releases A threat campaign package is compatible with the NGINX Plus release supported during the time the threat campaign package was released and with all future releases from that point in time on. In other words, it is not compatible with earlier App Protect DoS releases. Those older releases are not supported at this point in time so you will have to upgrade App Protect DoS to benefit from the support which includes Threat campaigns updates. ## Upgrading App Protect DoS You can upgrade to the latest NGINX Plus and App Protect DoS versions by downloading and installing the latest F5 DoS for NGINX package. When upgrading from this package, App Protect DoS will be uninstalled and reinstalled. The old default security policy is deleted and the new default security policy is installed. If you have created a custom security policy, the policy persists and you will need to update `nginx.conf` and point to the custom security policy by referencing the json file (using the full path). If you upgrade your NGINX version outside the App Protect DoS module, App Protect DoS will be uninstalled and you will need to reinstall it. You need to restart NGINX after an upgrade. ## SELinux The default settings for Security-Enhanced Linux (SELinux) on modern Red Hat Enterprise Linux (RHEL) and related Linux distributions can be very strict, erring on the side of security rather than convenience. Although the App Protect DoS applies its SELinux policy module during installation, your specific configuration might be blocked unless you adjust the policy or modify file labels. ### Modifying File Labels For example, if you plan to store your log configuration files in `/etc/logs` - you should change the default SELinux file context for this directory: ```shell semanage fcontext -a -t httpd_config_t /etc/logs restorecon -Rv /etc/logs ``` ### Syslog to Custom Port If you want to send logs to some unreserved port, you can use semanage to add the desired port (here, 35514) to the syslogd_port_t type: ```shell semanage port -a -t syslogd_port_t -p tcp 35514 ``` Review the syslog ports by entering the following command: ```shell semanage port -l | grep syslog ``` ## Kubernetes Deployment Examples ### App Protect DoS `appprotect-dos.yaml`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: appprotect-dos namespace: appprotect-dos-wp-diff labels: app: appprotect-dos spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: appprotect-dos strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 template: metadata: labels: app: appprotect-dos spec: containers: - name: ubuntu-bados image: example.com/ubuntu_app_protect_dos:latest imagePullPolicy: Always resources: requests: cpu: "200m" memory: "500Mi" limits: cpu: "900m" memory: "800Mi" ports: - containerPort: 80 name: web - containerPort: 8090 name: probe - containerPort: 8091 name: probe500 livenessProbe: httpGet: path: /app_protect_dos_liveness port: 8090 initialDelaySeconds: 0 periodSeconds: 10 readinessProbe: httpGet: path: /app_protect_dos_readiness port: 8090 initialDelaySeconds: 0 periodSeconds: 10 volumeMounts: - name: shared mountPath: /shared/ - name: conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf - name: root-script mountPath: /root/entrypoint.sh subPath: entrypoint.sh - name: log-default mountPath: /etc/app_protect_dos/log-default.json subPath: log-default.json volumes: - name: shared persistentVolumeClaim: claimName: pvc-appprotect-dos-shared - name: conf configMap: name: cm-appprotect-dos-nginx items: - key: nginx.conf path: nginx.conf - name: root-script configMap: name: cm-appprotect-dos-entry defaultMode: 0755 items: - key: entrypoint.sh path: entrypoint.sh - name: log-default configMap: name: cm-appprotect-dos-log-default defaultMode: 0755 items: - key: log-default.json path: log-default.json ``` `svc-appprotect-dos.yaml`: ```yaml apiVersion: v1 kind: Service metadata: name: svc-appprotect-dos namespace: appprotect-dos-wp-diff labels: app: appprotect-dos spec: ports: - name: app port: 80 protocol: TCP nodePort: 80 selector: app: appprotect-dos type: NodePort ``` `log-default.json`: ```json { "filter": { "traffic-mitigation-stats": "all", "bad-actors": "all", "attack-signatures": "all" } } ``` `entrypoint.sh`: ```shell #!/usr/bin/env bash USER=nginx LOGDIR=/var/log/adm # prepare environment mkdir -p /var/run/adm /tmp/cores ${LOGDIR} chmod 755 /var/run/adm /tmp/cores ${LOGDIR} chown ${USER}:${USER} /var/run/adm /tmp/cores ${LOGDIR} # run processes /bin/su -s /bin/bash -c "/usr/bin/adminstall > ${LOGDIR}/adminstall.log 2>&1" ${USER} /usr/sbin/nginx -g 'daemon off;' & /bin/su -s /bin/bash -c "/usr/bin/admd -d --log info > ${LOGDIR}/admd.log 2>&1 &" ${USER} ``` `install.sh`: ```shell #!/bin/bash set -ex DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" kubectl -n appprotect-dos-wp-diff create cm cm-appprotect-dos-nginx --from-file ${DIR}/nginx.conf kubectl -n appprotect-dos-wp-diff create cm cm-appprotect-dos-entry --from-file ${DIR}/entrypoint.sh kubectl -n appprotect-dos-wp-diff create cm cm-appprotect-dos-log-default --from-file ${DIR}/log-default.json kubectl create -f ${DIR}/appprotect-dos.yaml #kubectl create -f ${DIR}/svc-appprotect-dos.yaml ``` `nginx.conf`: ```nginx user nginx; worker_processes 1; error_log /var/log/nginx/error.log debug; worker_rlimit_nofile 65535; working_directory /tmp/cores; load_module modules/ngx_http_app_protect_dos_module.so; events { worker_connections 65535; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format log_napd ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, ' 'outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ' 'ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; app_protect_dos_security_log_enable on; app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" /var/log/adm/logger.log; # app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=1.2.3.4:5261; app_protect_dos_liveness on; # uri:/app_protect_dos_liveness port:8090 app_protect_dos_readiness on; # uri:/app_protect_dos_readiness port:8090 server { listen 80 reuseport; server_name serv; set $loggable '0'; access_log /var/log/nginx/access.log log_napd if=$loggable; # access_log syslog:server=1.1.1.1:5561 log_napd if=$loggable; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; location / { app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_monitor uri=http://serv:80/ protocol=http1; proxy_pass http://1.2.3.4:80; } } server { listen 8090; server_name probe; location / { proxy_pass http://localhost:8091; } } server { listen 8091; return 503; } sendfile on; tcp_nopush on; keepalive_timeout 65; } ``` ### App Protect DoS Arbitrator Arbitrator (arb) is an internal service that is essential for the scaling scenarios. The arbitrator service should be deployed in the same namespace as F5 DoS for NGINX. `appprotect-dos-arb.yaml`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: appprotect-dos-arb namespace: appprotect-dos-wp-diff spec: replicas: 1 selector: matchLabels: app: appprotect-dos-arb template: metadata: labels: app: appprotect-dos-arb spec: containers: - name: arb-svc image: example.com/app_protect_dos_arb:latest resources: requests: cpu: "200m" memory: "500Mi" limits: cpu: "900m" memory: "800Mi" ports: - containerPort: 3000 ``` `svc-appprotect-dos-arb.yaml`: ```yaml apiVersion: v1 kind: Service metadata: name: svc-appprotect-dos-arb namespace: appprotect-dos-wp-diff spec: selector: app: appprotect-dos-arb ports: - name: arb port: 3000 protocol: TCP targetPort: 3000 clusterIP: None ``` `install_appprotect-arb.sh`: ```shell #!/bin/bash set -ex DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" kubectl -n appprotect-dos-wp-diff apply -f ${DIR}/appprotect-dos-arb.yaml kubectl -n appprotect-dos-wp-diff apply -f ${DIR}/svc-appprotect-dos-arb.yaml ``` `install F5 DoS for NGINX with ARB service`: ```shell #!/bin/bash set -ex DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" kubectl create ns appprotect-dos-wp-diff ${DIR}/appprotect-dos/install.sh ${DIR}/appprotect-dos-arb/install_appprotect-dos-arb.sh ``` --- ## App Protect DoS eBPF manager ### Overview The eBPF Manager is a powerful and efficient tool designed to simplify and secure the deployment of eBPF (Extended Berkeley Packet Filter) programs for advanced networking use cases. Its primary responsibilities include program installation and managing client interactions to enable real-time packet processing and mitigation solutions. ### CLI Options for Flexible Configuration The eBPF Manager comes with configurable command-line flags for ease of use and deployment customization. Key options include: * Interface Selection: * -i, --interface [interfaces...]: Specify one or more network interfaces for eBPF XDP program deployment. If omitted, it defaults to all non-virtual, active network devices. * gRPC UDS Ownership: * -u, --user : Set the user ownership for the gRPC Unix Domain Socket (UDS). Defaults to nginx. * -g, --group : Set the group ownership for the gRPC Unix Domain Socket (UDS). Defaults to nginx. -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.3" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.3/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.3. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.3 December 19, 2023 In this release, F5 DoS for NGINX supports NGINX Plus R31. ### New Features - Support for Nginx Plus R31 - [Support for Alpine 3.17](learn-about-deployment.md#alpine-315x--317x-installation) ### Supported Packages #### App Protect DoS ##### Alpine 3.17 - app-protect-dos-31.4.3.1-r1.apk ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-31.4.3.1-1.el7.ngx.x86_64.rpm ##### RHEL 8 and Rocky Linux 8 - app-protect-dos-31.4.3.1-1.el8.ngx.x86_64.rpm ##### Debian 11 - app-protect-dos_31.4.3.1-1~bullseye_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_31.4.3.1-1~focal_amd64.deb ##### Ubuntu 22.04 - app-protect-dos_31.4.3.1-1~jammy_amd64.deb #### NGINX Plus - NGINX Plus R31 ### Important Note - Starting with this release, Alpine 3.15 support has been deprecated. -------------------------------------------------------------------------------- title: "Best Practices" url: https://docs.nginx.com/nginx-app-protect-dos/deployment-guide/best-practices/index.md description: F5 DoS for NGINX Best Practices Deployment. -------------------------------------------------------------------------------- This guide shows how to modify your NGINX configuration to enable F5 DoS for NGINX (NGINX App Protect DoS). We will configure F5 DoS For NGINX to protect a proxy server. ## F5 DoS Configuration ### Load Module ```nginx load_module modules/ngx_http_app_protect_dos_module.so; ``` ### Enable Add the directive in the appropriate context, You can set it in location, server, or http blocks: ```nginx app_protect_dos_enable on; ``` #### Set a Protected Object Name Choose a unique name. You can set it in location, server, or http blocks. ```nginx app_protect_dos_name po-example; ``` **Note**: Although optional, we strongly recommend specifying a name for each Protected Object (PO) to improve organization and maintainability. If no name is provided, the virtual server is assigned an auto-generated name using the following syntax: ```nginx line_number-server_name:seq-location_name Example: 30-backend:1-/abc ``` Where: - `line number:` the line number of the server block (`server {`) in the `nginx.conf` file (i.e. `30`)
- `server name:` taken from directive `server_name` (i.e. `backend`)
seq: 0 for server block, increments for each location block. i.e. VS created from server block will have 0 and VS's from location blocks will be 1,2,3,... (i.e. `1`) - `location name:` the name of the location (i.e. `/abc`) Capacity limits - Up to 300 Protected Objects in versions up to 4.3
- Up to 1,000 Protected Objects in versions 4.4 and later
### Set a Monitor directive The `app_protect_dos_monitor` directive monitors the stress level of a Protected Object by generating requests from localhost (127.0.0.1) that traverse your NGINX configuration like normal client traffic (through the same `server` / `location` / `proxy_pass` chain). **This directive is mandatory for optimal accuracy (it may be omitted only when using HTTP/1.1, though it is still strongly recommended).** ```nginx app_protect_dos_monitor uri= [protocol=http1|http2|grpc|websocket] [timeout=] [proxy_protocol=on|off]; ``` - `uri` is the value of `server_name`, optionally followed by `:port`, and then the location path. Examples: `my_server/`, `example_server:81/abc` A complete guide on configuring the Monitor Directive can be found here: [Monitor Directive](https://docs.nginx.com/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/#monitor-directive-app_protect_dos_monitor). **Monitor directive best practice** - Monitor the same virtual host and path that your users hit. Set `uri=` to the `server_name[:port]/path` that matches the `server_name` and `listen` directives, **not** to the upstream IP:port.
Examples:
For `server_name "my_server"` on port `80` and path `/` (port 80 is default, so it can be omitted): ```nginx app_protect_dos_monitor uri=my_server/; ``` For `server_name "serv"` on port `81` with location path `/abc`: ```nginx app_protect_dos_monitor uri=serv:81/abc protocol=http1 timeout=7; ``` A full example with upstream:
```nginx upstream backend { server 10.197.24.136:3000; } server { listen 80 reuseport; server_name example_srv; location / { app_protect_dos_enable on; app_protect_dos_name "main_app"; # ✅ Good: monitor hits NGINX using server_name and is proxied to the upstream app_protect_dos_monitor uri=example_srv:80/ protocol=http1 timeout=7; # ❌ Bad: do NOT point the monitor directly at the upstream IP # app_protect_dos_monitor uri=10.197.24.136:3000/ protocol=http1 timeout=7; proxy_pass http://backend; } } ``` - Avoid monitors that short-circuit upstreams (for example, `return 200` locally); this will under-estimate stress.
- Choose `timeout` slightly above your upstream’s p95/p99 latency under normal load, but low enough to react quickly under stress.
- Monitor traffic originates from `127.0.0.1`. Exclude it from rate and connection limits as needed.
- Define the monitor inside each protected `location` block.
## Arbitrator It is required when more than one F5 DoS for NGINX instance is deployed. Its primary function is to ensure that all instances are aware of—and share—the same state for each Protected Object.
A complete guide on configuring F5 DoS for NGINX Arbitrator be found here: [F5 DoS for NGINX Arbitrator](https://docs.nginx.com/nginx-app-protect-dos/deployment-guide/learn-about-deployment/#f5-dos-for-nginx-arbitrator)
Enable the F5 DoS for NGINX Arbitrator in the `http` context of the `nginx.conf` file: ```nginx app_protect_dos_arb_fqdn 10.1.10.22; ``` ## EBPF manager The eBPF Manager is a high-performance component that simplifies and secures the deployment of eBPF (Extended Berkeley Packet Filter) programs for advanced networking use cases. Enable the L4-accelerated mitigation feature in the http context of the nginx.conf file: ```nginx app_protect_dos_accelerated_mitigation on; ``` ## ELK Dashboards ELK stands for Elasticsearch, Logstash, and Kibana. Logstash receives logs from F5 DoS, normalizes them, and stores them in the Elasticsearch index. Kibana allows you to visualize and navigate the logs using purpose-built dashboards.
A complete guide on configuring ELK can be found here: [F5 DoS for NGINX ELK Dashboards](https://github.com/f5devcentral/nap-dos-elk-dashboards)
F5 DoS directives should appear in your `nginx.conf` as shown. Replace `ip_kibana` with the hostname of the server running your ELK Docker container:
```nginx http { log_format log_dos ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; ... server { ... app_protect_dos_security_log_enable on; app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=ip_kibana:5261; location / { app_protect_dos_enable on; set $loggable '0'; access_log syslog:server=ip_kibana:5561 log_dos if=$loggable; ... } } ... } ``` ## Live Activity Monitoring F5 DoS for NGINX provides a range of application monitoring tools: - F5 DoS for NGINX Dashboard: A dynamic interface for real-time monitoring and detailed views of Protected Objects.
- F5 DoS for NGINX REST API: An interface that exposes comprehensive metrics for Protected Objects.
A complete guide on configuring F5 DoS for NGINX Live Activity Monitoring be found here: [F5 DoS for NGINX Live Activity Monitoring](https://docs.nginx.com/nginx-app-protect-dos/monitoring/live-activity-monitoring/)
Below is an example configuration that limits API location access to the local network using the allow and deny directives, and uses HTTP Basic Authentication to restrict the PATCH, POST, and DELETE methods to specific users.
To view the dashboard, enter its address in your browser’s address bar. For example, http://192.168.1.23/dashboard-dos.html displays the dashboard page located in `/usr/share/nginx/html`, as specified by the root directive.
```nginx http { # ... server { listen 192.168.1.23; # ... location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } app_protect_dos_api; allow 192.168.1.0/24; deny all; } location = /dashboard-dos.html { root /usr/share/nginx/html; } } } ``` ## Example nginx.conf ```nginx user nginx; worker_processes auto; error_log /var/log/nginx/error.log error; load_module modules/ngx_http_app_protect_dos_module.so; events { worker_connections 1024; } http { app_protect_dos_arb_fqdn 10.1.10.22; app_protect_dos_accelerated_mitigation on; sendfile on; tcp_nopush on; keepalive_timeout 65; log_format log_dos ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, ' 'outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ' 'ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; server { listen 80 reuseport; server_name example_srv; access_log /var/log/nginx/access.log log_dos if=$loggable; app_protect_dos_security_log_enable on; app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=10.197.30.219:5261; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; location / { app_protect_dos_enable on; app_protect_dos_name "main_app"; set $loggable '0'; access_log syslog:server=10.97.30.219:5561 log_dos if=$loggable; app_protect_dos_monitor uri=example_srv:80/ protocol=http1 timeout=7; proxy_pass http://10.197.24.136:3000; } } server { listen 800; location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } app_protect_dos_api; allow 192.168.1.0/24; deny all; } location = /dashboard-dos.html { root /usr/share/nginx/html; } } } ``` -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.2" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.2/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.2. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.2 August 15, 2023 In this release, F5 DoS for NGINX supports NGINX Plus R30. ### New Features - Support for Nginx Plus R30 - [Support for Ubuntu 22.04](learn-about-deployment.md#debian--ubuntu-installation) - Support for HTTP3/QUIC - Improvement of Embedded Server Health mechanism ### Supported Packages #### App Protect DoS ##### Alpine 3.15 - app-protect-dos-30.4.2.0-r1.apk ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-30.4.2.0-1.el7.ngx.x86_64.rpm ##### RHEL 8 and Rocky Linux 8 - app-protect-dos-30.4.2.0-1.el8.ngx.x86_64.rpm ##### Debian 11 - app-protect-dos_30.4.2.0-1~bullseye_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_30.4.2.0-1~focal_amd64.deb ##### Ubuntu 22.04 - app-protect-dos_30.4.2.0-1~jammy_amd64.deb #### NGINX Plus - NGINX Plus R30 ### Important Notes - Installing L4 accelerated mitigation feature (install `app-protect-dos-ebpf`) configures `nginx` and `admd` to run with root privileges. - Support for `proxy_protocol` configuration: `proxy_protocol` monitor parameter should be used when the `listen` directive of the correspondent server block contains the `proxy_protocol` parameter. - If F5 WAF for NGINX is installed, app protect should be disabled for the location of DoS Live Activity Monitoring API. For example: ```shell location /api { app_protect_enable off; app_protect_dos_api; } ``` - Port configuration in `app_protect_dos_monitor` should correspond to the port, the server listens to. Misconfiguration can potentially cause a false attack declaration. For example: ```shell server { listen 8080; server_name myservice.com; location / { app_protect_dos_monitor "myservice.com:8080/"; } } ``` - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for the attack to be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. - The package dependencies for F5 DoS for NGINX have changed in this release, replacing the `curl` dependencies with `libcurl` only. For more information, see the [F5 DoS for NGINX Deployment Guide](/nap-dos/deployment-guide/learn-about-deployment.md#prerequisites). - Starting with this release, Ubuntu 18.04 support has been deprecated. -------------------------------------------------------------------------------- title: "Kubernetes" url: https://docs.nginx.com/nginx-app-protect-dos/deployment-guide/kubernetes/index.md description: Install F5 DoS for NGINX on Kubernetes using manifests to deploy DoS protection as a sidecar container alongside NGINX Plus. -------------------------------------------------------------------------------- This guide explains how to install F5 DoS for NGINX on Kubernetes. It covers the common steps for any Kubernetes-based deployment, then provides the manifest-based installation steps. ## Before you begin Before you start, make sure you have: - A functional Kubernetes cluster - An active F5 DoS for NGINX subscription (purchased or trial) - [Docker](https://docs.docker.com/get-started/get-docker/) To review supported operating systems, read the [Releases](/nap-dos/releases) topic. ## Download your subscription credentials Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. ## Create a Dockerfile In the same folder as your credential files, create a _Dockerfile_ based on your desired operating system image using an example from the following sections. As written, the examples install the most recent version of F5 DoS for NGINX and NGINX Plus. To build a specific version instead, set the `DOS_VERSION` build argument, using the syntax shown in the comment at the top of the Dockerfile for your operating system. On Debian and Ubuntu, also set `NGINX_PLUS_VERSION`, because `apt` considers only the newest `nginx-plus` available and does not select an older one to satisfy the module's dependency. To review which NGINX Plus release each version requires, read the [Releases](/nap-dos/releases) topic. ### Alpine Linux #### NGINX Plus ```dockerfile # syntax=docker/dockerfile:1 # Supported OS_VER's are 3.21/3.22 ARG OS_VER="3.22" # Base image FROM alpine:${OS_VER} # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="=37+4.9.6-r1" # NGINX Plus needs no pin here, because apk can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/apk/cert.pem,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/apk/cert.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub \ && printf "https://pkgs.nginx.com/plus/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories \ && printf "https://pkgs.nginx.com/app-protect-dos/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories \ && apk update \ && apk add "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log \ && rm -rf /var/cache/apk/* # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Amazon Linux #### NGINX Plus ```dockerfile # For AmazonLinux 2023: FROM amazonlinux:2023 # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-amazonlinux2023.repo https://cs.nginx.com/static/files/plus-amazonlinux2023.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo https://cs.nginx.com/static/files/app-protect-dos-amazonlinux2023.repo \ && dnf install -y "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && dnf clean all \ && rm -rf /var/cache/dnf \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Debian #### NGINX Plus ```dockerfile # Where can be bullseye/bookworm/trixie FROM debian:bullseye # Leave both arguments empty to install the most recent version of each package. # To build a specific version, set both, for example: # --build-arg DOS_VERSION="=37+4.9.6-1~bullseye" --build-arg NGINX_PLUS_VERSION="=37.0.*-1~bullseye" # Pin NGINX Plus as well: apt considers only the newest nginx-plus available and # does not select an older one to satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" ARG NGINX_PLUS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update \ && DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 debian-archive-keyring \ && mkdir -p /etc/ssl/nginx/ /etc/nginx/ \ && wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/debian $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/debian $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \ && apt-get update \ && DEBIAN_FRONTEND="noninteractive" apt-get install -y "app-protect-dos${DOS_VERSION}" "nginx-plus-module-appprotectdos${DOS_VERSION}" "nginx-plus${NGINX_PLUS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && apt-get remove --purge --auto-remove -y apt-transport-https lsb-release gnupg2 wget \ && rm -rf /var/lib/apt/lists/* \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### RHEL 8 #### NGINX Plus ```dockerfile # For UBI 8 FROM registry.access.redhat.com/ubi8 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} \ && subscription-manager refresh \ && subscription-manager attach --auto || true \ && subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms \ && subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-8.repo https://cs.nginx.com/static/files/plus-8.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-8.repo https://cs.nginx.com/static/files/app-protect-dos-8.repo \ && dnf -y install "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && rm /etc/yum.repos.d/plus-8.repo \ && rm /etc/yum.repos.d/app-protect-dos-8.repo \ && dnf clean all \ && rm -rf /var/cache/yum \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### RHEL 9 #### NGINX Plus ```dockerfile # For UBI 9 FROM registry.access.redhat.com/ubi9 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} \ && subscription-manager refresh \ && subscription-manager attach --auto || true \ && subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms \ && subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-9.repo https://cs.nginx.com/static/files/plus-9.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo \ && dnf -y install "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && rm /etc/yum.repos.d/plus-9.repo \ && rm /etc/yum.repos.d/app-protect-dos-9.repo \ && dnf clean all \ && rm -rf /var/cache/yum \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### RHEL 10 #### NGINX Plus ```dockerfile # For UBI 10 FROM registry.access.redhat.com/ubi10 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} \ && subscription-manager refresh \ && subscription-manager attach --auto || true \ && subscription-manager repos --enable=rhel-10-for-x86_64-baseos-rpms \ && subscription-manager repos --enable=rhel-10-for-x86_64-appstream-rpms \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-10.repo https://cs.nginx.com/static/files/plus-10.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-10.repo https://cs.nginx.com/static/files/app-protect-dos-10.repo \ && dnf -y install "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && rm /etc/yum.repos.d/plus-10.repo \ && rm /etc/yum.repos.d/app-protect-dos-10.repo \ && dnf clean all \ && rm -rf /var/cache/yum \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Rocky Linux 9 #### NGINX Plus ```dockerfile # syntax=docker/dockerfile:1 # For Rocky Linux 9 FROM rockylinux:9 # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX: RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf -y install ca-certificates epel-release 'dnf-command(config-manager)' \ && curl -o /etc/yum.repos.d/plus-9.repo https://cs.nginx.com/static/files/plus-9.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo \ && dnf config-manager --set-enabled crb \ && dnf install -y "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && dnf clean all \ && rm -rf /var/cache/dnf \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ### Ubuntu #### NGINX Plus ```dockerfile # syntax=docker/dockerfile:1 # For Ubuntu # Where version can be: jammy/noble FROM ubuntu:noble # Leave both arguments empty to install the most recent version of each package. # To build a specific version, set both, for example: # --build-arg DOS_VERSION="=37+4.9.6-1~noble" --build-arg NGINX_PLUS_VERSION="=37.0.*-1~noble" # Pin NGINX Plus as well: apt considers only the newest nginx-plus available and # does not select an older one to satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" ARG NGINX_PLUS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update \ && apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 ubuntu-keyring \ && mkdir -p /etc/ssl/nginx/ /etc/nginx/ \ && wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/ubuntu $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/ubuntu $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \ && apt-get update \ && DEBIAN_FRONTEND="noninteractive" apt-get install -y "app-protect-dos${DOS_VERSION}" "nginx-plus-module-appprotectdos${DOS_VERSION}" "nginx-plus${NGINX_PLUS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && apt-get remove --purge --auto-remove -y apt-transport-https lsb-release gnupg2 wget && rm -rf /var/lib/apt/lists/* \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` ## Create DOS entrypoint.sh Docker startup script which spins up all App Protect DoS processes, must have executable permissions ```shell #!/usr/bin/env bash USER=nginx LOGDIR=/var/log/adm # prepare environment mkdir -p /var/run/adm /tmp/cores ${LOGDIR} chmod 755 /var/run/adm /tmp/cores ${LOGDIR} chown ${USER}:${USER} /var/run/adm /tmp/cores ${LOGDIR} # run processes /bin/su -s /bin/bash -c "/usr/bin/adminstall > ${LOGDIR}/adminstall.log 2>&1" ${USER} /bin/su -s /bin/bash -c "/usr/bin/admd -d --log info > ${LOGDIR}/admd.log 2>&1 &" ${USER} /usr/sbin/nginx -g 'daemon off;' ``` ## Build the Docker image Your folder should contain the following files: - _nginx-repo.crt_ - _nginx-repo.key_ - _entrypoint.sh_ - _nginx.conf_ - _Dockerfile_ To build an image, use the following command, replacing `` as appropriate: ```shell sudo docker build --no-cache --platform linux/amd64 \ --secret id=nginx-crt,src=nginx-repo.crt \ --secret id=nginx-key,src=nginx-repo.key \ -t . ``` To build a specific version instead, add the two build arguments. For example, on Debian 11: ```shell sudo docker build --no-cache --platform linux/amd64 \ --secret id=nginx-crt,src=nginx-repo.crt \ --secret id=nginx-key,src=nginx-repo.key \ --build-arg DOS_VERSION="=37+4.9.6-1~bullseye" \ --build-arg NGINX_PLUS_VERSION="=37.0.*-1~bullseye" \ -t . ``` Once you have built the image, push it to your private image repository, which must be accessible to your Kubernetes cluster. ## Use Manifests to install F5 DOS for NGINX The `` argument must be the _contents_ of the file, not the file itself. Ensure there are no additional characters such as extra whitespace. ### Create Manifest files The default configuration provided creates two replicas, each hosting NGINX and DOS services together in a single Kubernetes pod. Create all of these files in a single folder (Such as `/manifests`). On manifest deployment, set the environment variables for image repository and tag. Set environment variable `DOS_IMAGE_REPOSITORY` with your actual nginx-dos image name. Set environment variable `DOS_IMAGE_TAG` with your actual nginx-dos image tag. ```dos-namespace.yaml apiVersion: v1 kind: Namespace metadata: name: app-protect-dos ``` ```dos-nginx-conf-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: dos-nginx-conf namespace: app-protect-dos data: nginx.conf: | user nginx; worker_processes auto; error_log /var/log/nginx/error.log error; worker_rlimit_nofile 65535; load_module modules/ngx_http_app_protect_dos_module.so; working_directory /tmp/cores; events { worker_connections 65535; } http { app_protect_dos_arb_fqdn svc-appprotect-dos-arb.arb.svc.cluster.local; sendfile on; tcp_nopush on; keepalive_timeout 65; log_format log_dos ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, ' 'outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ' 'ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; # Health endpoints for probes app_protect_dos_liveness on; # uri:/app_protect_dos_liveness port:8090 app_protect_dos_readiness on; # uri:/app_protect_dos_readiness port:8090 server { listen 8090; server_name probe; location / { proxy_pass http://localhost:8091; } } server { listen 8091; return 503; } server { listen 80 reuseport; server_name serv; proxy_http_version 1.1; access_log /var/log/nginx/access.log log_dos if=$loggable; app_protect_dos_security_log_enable on; app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=:5261; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; location / { app_protect_dos_enable on; app_protect_dos_name "main_serv"; app_protect_dos_monitor uri=http://serv:80/ protocol=http1; proxy_pass http://127.0.0.1/proxy$request_uri; } location /proxy { app_protect_dos_enable off; client_max_body_size 0; default_type text/html; return 200 "Hello! I got your URI request - $request_uri\n"; } } } ``` ```dos-log-default-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: dos-log-default namespace: app-protect-dos data: log-default.json: | { "filter": { "traffic-mitigation-stats": "all", "bad-actors": "all", "attack-signatures": "all" } } ``` ```dos-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: app-protect-dos namespace: app-protect-dos labels: app: app-protect-dos spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: app-protect-dos strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 template: metadata: labels: app: app-protect-dos spec: containers: - name: nginx-app-protect-dos image: ${DOS_IMAGE_REPOSITORY}:${DOS_IMAGE_TAG} imagePullPolicy: Always command: ["/bin/bash", "-c"] args: - | /root/entrypoint.sh resources: requests: cpu: "200m" memory: "500Mi" limits: cpu: "900m" memory: "800Mi" ports: - containerPort: 80 name: web - containerPort: 8090 name: probe - containerPort: 8091 name: probe500 livenessProbe: httpGet: path: /app_protect_dos_liveness port: 8090 initialDelaySeconds: 5 periodSeconds: 10 readinessProbe: httpGet: path: /app_protect_dos_readiness port: 8090 initialDelaySeconds: 5 periodSeconds: 10 volumeMounts: - name: shared-dir mountPath: /shared/ - name: conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf - name: log-default mountPath: /etc/app_protect_dos/log-default.json subPath: log-default.json - name: license-token-volume mountPath: /etc/nginx/license.jwt subPath: license.jwt readOnly: true volumes: - name: shared-dir emptyDir: { } - name: conf configMap: name: dos-nginx-conf items: - key: nginx.conf path: nginx.conf - name: log-default configMap: name: dos-log-default defaultMode: 0644 items: - key: log-default.json path: log-default.json - name: license-token-volume secret: secretName: license-token items: - key: license.jwt path: license.jwt ``` ```dos-service.yaml apiVersion: v1 kind: Service metadata: name: nap-dos namespace: app-protect-dos spec: ports: - name: app port: 80 protocol: TCP selector: app: app-protect-dos type: NodePort ``` ### Start the Manifest deployment From the folder containing the YAML files from the previous step (Suggested as `/manifests`), deploy F5 DOS for NGINX using `kubectl`: ```shell export DOS_IMAGE_REPOSITORY= export DOS_IMAGE_TAG= kubectl apply -f manifests/dos-namespace.yaml kubectl create secret generic license-token --from-file=license.jwt=license.jwt --type=nginx.com/license --namespace app-protect-dos kubectl apply -f dos-manifest/dos-log-default-configmap.yaml kubectl apply -f dos-manifest/dos-nginx-conf-configmap.yaml kubectl apply -f manifests/dos-deployment.yaml kubectl apply -f manifests/dos-service.yaml ``` It will apply all the configuration defined in the files to your Kubernetes cluster. You can then check the status of the deployment with `kubectl get`: ```shell kubectl --namespace app-protect-dos get deployments kubectl --namespace app-protect-dos get pods kubectl --namespace app-protect-dos get services ``` You should see output similar to the following: ```text ~$ kubectl --namespace app-protect-dos get deployments NAME READY UP-TO-DATE AVAILABLE AGE app-protect-dos 1/1 1 1 1m ~$ kubectl --namespace app-protect-dos get pods NAME READY STATUS RESTARTS AGE app-protect-dos-586fb94947-8sjnc 1/1 Running 0 1m ~$ kubectl --namespace app-protect-dos get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nap-dos LoadBalancer 10.43.83.225 80:30307/TCP 1m ``` ## Post-installation checks At this stage, you have finished deploying F5 DOS for NGINX. You can log in to the app-protect-dos pod using the following command: ```text kubectl exec -it app-protect-dos-586fb94947-8sjnc -n app-protect-dos -c nginx-app-protect-dos -- bash ``` Then run the following post-installation checks. You can run the following commands to ensure that F5 DoS for NGINX enforcement is operational. 1. Check that the three processes needed for F5 DoS for NGINX are running using `ps aux`: - admd - nginx: master process - nginx: worker process ```shell USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND nginx 7759 0.0 0.0 113120 1200 ? Ss Sep06 0:00 /bin/sh -c /usr/bin/admd -d --log info > /var/log/adm/admd.log 2>&1 root 7765 0.0 0.0 87964 1464 ? Ss Sep06 0:00 nginx: master process /usr/sbin/nginx -g daemon off; nginx 7767 0.0 0.1 615868 8188 ? Sl Sep06 0:04 nginx: worker process ``` 2. Verify that there are no NGINX errors in the `/var/log/nginx/error.log` and that the policy compiled successfully: ```shell 2025/12/07 09:14:34 [notice] 675#675: APP_PROTECT_DOS { "event": "shared_memory_connected", "worker_pid": 675, "mode": "operational", "mode_changed": true } 2025/12/07 09:14:34 [notice] 675#675: using the "epoll" event method 2025/12/07 09:14:34 [notice] 675#675: APP_PROTECT_DOS { "event": "configuration_load_success", "software_version": "36+4.8.3-1.el8.ngx"} 2025/12/07 09:14:34 [notice] 675#675: nginx/1.29.3 (nginx-plus-r36) 2025/12/07 09:14:34 [notice] 675#675: built by gcc 8.5.0 20210514 (Red Hat 8.5.0-28) (GCC) 2025/12/07 09:14:34 [notice] 675#675: OS: Linux 6.8.0-88-generic 2025/12/07 09:14:34 [notice] 675#675: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2025/12/07 09:14:34 [notice] 675#675: start worker processes 2025/12/07 09:14:34 [notice] 675#675: start worker process 679 2025/12/07 09:14:34 [notice] 679#679: APP_PROTECT_DOS { "event": "shared_memory_connected", "worker_pid": 679, "mode": "operational", "mode_changed": true } ``` 3. Verify that when you simulate an attack, attacker IP addresses are blocked while legitimate traffic passes through: a. Simulate good traffic: ```shell echo "Start Good Traffic 2" while true; do curl ${VS}/good1 & curl ${VS}/good2 & curl ${VS}/good3 & curl ${VS}/good4 sleep 0.1 done & ``` b. After 7 minutes start the attack: ```shell while [ true ] do ab -B ${BAD_IP1} -l -r -n 1000000 -c 150 -d -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ & ab -B ${BAD_IP2} -l -r -n 1000000 -c 150 -d -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ & ab -B ${BAD_IP3} -l -r -n 1000000 -c 150 -d -s 10 -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ killall ab done ``` c. Verify that legitimate traffic continues as usual while the attack traffic is blocked. 4. For DOS with L4 accelerated mitigation enabled Check that the ebpf_manager_dos process needed for F5 DoS for NGINX is running using `ps aux | grep /usr/bin/ebpf_manager_dos`: ``` root 1 0.0 0.0 4324 3072 ? Ss 19:32 0:00 bash -c /usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log root 7 0.2 0.0 1722732 14208 ? Sl 19:32 0:01 /usr/bin/ebpf_manager_dos root 46 0.0 0.0 3528 1792 pts/0 S+ 19:44 0:00 grep --color=auto /usr/bin/ebpf_manager_dos ``` Verify that there are no errors in the `/shared/ebpf_dos.log` and that the XDP program uploaded successfully: ```[2025-12-02 19:32:12] INFO: Uninstall old eBPF maps and XDP program [2025-12-02 19:32:13] INFO: Install eBPF maps and XDP program [2025-12-02 19:32:13] INFO: Start ebpf manager [2025-12-02 19:32:13] INFO: Version: 36+4.8.3-1~noble [2025-12-02 19:32:13] INFO: Start Periodic task for update time [2025-12-02 19:32:13] INFO: Owner of the UDS has been changed to user nginx and group nginx. [2025-12-02 19:32:13] INFO: Permissions of the UDS have been changed successfully for user nginx and group nginx. [2025-12-02 19:32:13] INFO: Async Callback Server listening on unix:/shared/ebpf_manager_dos_uds ``` To check F5 WAF for NGINX alongside F5 DoS for NGINX, just perform the normal tests as specified at [Admin Guide](https://docs.nginx.com/waf/install/virtual-environment/#post-installation-checks) ## F5 DoS for NGINX arbitrator ## F5 DoS for NGINX Arbitrator ### Overview F5 DoS for NGINX Arbitrator orchestrates all running F5 DoS for NGINX instances to synchronize local and global attack start and stop. F5 DoS for NGINX Arbitrator is a central coordinating component for managing multiple F5 DoS for NGINX instances in a network. It is needed when there is more than one F5 DoS for NGINX instance. Its primary function is to ensure that all instances are aware of and share the same state for each protected object. ### How the Arbitrator works - **Collecting state periodically**: The Arbitrator regularly collects state information from all running F5 DoS for NGINX instances. This collection occurs at set intervals, typically every 10 seconds. - **State initialization for new instances**: When a new F5 DoS for NGINX instance starts, it retrieves the initial state for each protected object from the Arbitrator rather than starting with an empty state. - **Updating state during an attack**: When an F5 DoS for NGINX instance detects an attack, it sends a notification to the Arbitrator. The Arbitrator updates the state of the affected protected object and propagates that state to all other instances. ### Why F5 DoS for NGINX Arbitrator is necessary F5 DoS for NGINX Arbitrator is essential for several reasons: - **Global state management**: Without the Arbitrator, each F5 DoS for NGINX instance manages its own isolated state for each protected object. This can lead to inconsistencies. For example, if instance A declares an attack on a protected object named "PO-Example," instance B remains unaware of it, potentially leaving the object vulnerable. - **Uniform attack detection**: With the Arbitrator, when instance A detects an attack on "PO-Example" and reports it, the Arbitrator updates the state of "PO-Example" and propagates it to all instances, including instance B. F5 DoS for NGINX Arbitrator maintains a consistent global state for protected objects across all F5 DoS for NGINX instances. This ensures attacks are detected and mitigated uniformly across your deployment. ### F5 DoS for NGINX Arbitrator Deployment 1. Pull the official F5 DoS for NGINX Arbitrator image with the command: ```shell docker pull docker-registry.nginx.com/nap-dos/app_protect_dos_arb:latest ``` 2. Create a container based on this image, for example, `app-protect-dos-arb` container: ```shell docker run --name app_protect_dos_arb -p 3000:3000 -d docker-registry.nginx.com/nap-dos/app_protect_dos_arb ``` 3. Verify that the `app-protect-dos-arb` container is up and running with the `docker ps` command. 4. DNS records are required for F5 DoS for NGINX Arbitrator to work properly and be accessible by F5 DoS for NGINX servers. Ensure that the `svc-appprotect-dos-arb` or configured Arbitrator FQDN (with `app_protect_dos_arb_fqdn` directive) has a valid DNS resolution. This step is necessary only for VM/Docker deployments with arbitrator. When the arbitrator is in the same Kubernetes namespace as F5 DoS for NGINX, this step is not needed. ### Multi-VM Deployment The Arbitrator service is standalone. If it goes down, it can be restarted and immediately recovers all required information from F5 DoS for NGINX instances, which report to it every 10 seconds. Its downtime is around 10 to 20 seconds, which does not affect F5 DoS for NGINX operation. F5 DoS for NGINX Arbitrator connects to port 3000. All modules try to connect to it automatically. If it's not accessible, each instance operates in standalone mode. F5 DoS for NGINX does not support mutual TLS (mTLS) or password authentication between DoS servers and the Arbitrator. Arbitrator is not exposed outside the namespace. It is the customer's responsibility to isolate it from external access. This applies to all Arbitrator deployments, not only multi-VM. ## Next steps -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.1" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.1/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.1. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.1 May 2, 2023 In this release, F5 DoS for NGINX supports NGINX Plus R29. ### New Features - Support for NGINX Plus R29 - Support for Rocky Linux 8 ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-29+4.1.2-1.el7.ngx.x86_64.rpm ##### RHEL 8 and Rocky Linux 8 - app-protect-dos-29+4.1.2-1.el8.ngx.x86_64.rpm ##### Debian 11 - app-protect-dos_29+4.1.2-1~bullseye_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_29+4.1.2-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_29+4.1.2-1~focal_amd64.deb ##### Alpine 3.15 - app-protect-dos-29.4.1.2-r1.apk #### NGINX Plus - NGINX Plus R29 ### Important Notes - Installing L4 accelerated mitigation feature (install `app-protect-dos-ebpf`) configures `nginx` and `admd` to run with root privileges. - Support for `proxy_protocol` configuration: `proxy_protocol` monitor parameter should be used when the `listen` directive of the correspondent server block contains the `proxy_protocol` parameter. - If F5 WAF for NGINX is installed, app protect should be disabled for the location of DoS Live Activity Monitoring API. For example: ```shell location /api { app_protect_enable off; app_protect_dos_api; } ``` - Port configuration in `app_protect_dos_monitor` should correspond to the port, the server listens to. Misconfiguration can potentially cause a false attack declaration. For example: ```shell server { listen 8080; location / { app_protect_dos_monitor "myservice.com:8080"; } } ``` - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for the attack to be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Monitor directive `app_protect_dos_monitor` with `proxy_protocol` parameter can not be configured on Ubuntu 18.04. As a result, gRPC and HTTP/2 DoS protection for `proxy_protocol` configuration is not supported. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. - The package dependencies for F5 DoS for NGINX have changed in this release, replacing the `curl` dependencies with `libcurl` only. For more information, see the [F5 DoS for NGINX Deployment Guide](/nap-dos/deployment-guide/learn-about-deployment.md#prerequisites). -------------------------------------------------------------------------------- title: "Installing F5 WAF & DoS for NGINX AMIs on Amazon EC2" url: https://docs.nginx.com/nginx-app-protect-dos/deployment-guide/installing-nginx-plus-with-dos-and-waf-on-amazon-web-services/index.md description: Install F5 NGINX Plus, F5 WAF & DoS for NGINX Plus on Amazon Web Services (AWS), to provide sophisticated Layer 7 load balancing, Modern app security solution, behavioral DoS detection and mitigation that works seamlessly in DevOps environments for your apps running on AmazonLinux 2023, RHEL, Debian and Ubuntu Linux OS. -------------------------------------------------------------------------------- NGINX, Inc. participates in the Amazon Web Services (AWS) Partner Network as a Standard Technology Partner. We offer Amazon Machine Images (AMIs) for use in the Amazon Elastic Compute Cloud (EC2), available at the AWS Marketplace for several operating systems, including Amazon Linux, Red Hat Enterprise Linux, and Ubuntu. The AMIs contain combination of the following components: - Latest version of [F5 NGINX Plus](https://www.f5.com/products/nginx/nginx-plus), optimized for use on Amazon EC2 - Latest version of [F5 DoS for NGINX](https://docs.nginx.com/nginx-app-protect-dos/), optimized for use on Amazon EC2 - Latest version of [F5 WAF for NGINX](https://docs.nginx.com/nginx-app-protect-waf/), optimized for use on Amazon EC2 - Pre-packaged software for building highly available (HA) NGINX Plus configurations ## Install To quickly set up an environment with NGINX Plus, F5 WAF for NGINX and F5 DoS for NGINX on AWS: 1. Follow the instructions in [Getting Started with Amazon EC2 Linux Instances](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) to sign up on AWS and get more information about EC2 itself. 2. Proceed to the product page for the appropriate AMI at the AWS Marketplace, and launch the AMI. - [NGINX Plus with F5 DoS for NGINX Plus – AmazonLinux 2023 AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-x5nkvphsrg6ag) - [NGINX Plus with F5 DoS for NGINX Plus – RHEL8 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-e6bifer7o6uzm) - [NGINX Plus with F5 DoS for NGINX Plus – RHEL9 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-eoddgsaqujgkm) - [NGINX Plus with F5 DoS for NGINX Plus – Debian 11 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-56oveh2rsxsbq) - [NGINX Plus with F5 DoS for NGINX Plus – Ubuntu 22.04 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-l6f2q2ykrjufy) - [NGINX Plus with F5 DoS for NGINX Plus – Ubuntu 24.04 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-osz3wmirx5zqg) - [NGINX Plus with F5 WAF & DoS for NGINX Plus – RHEL 8 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-6pvnoyr2mp2co) - [NGINX Plus with F5 WAF & DoS for NGINX Plus – RHEL 9 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-kmtitev2hhrym) - [NGINX Plus with F5 WAF & DoS for NGINX Plus – Debian 11 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-wbyobl7a55vcu) - [NGINX Plus with F5 WAF & DoS for NGINX Plus – Ubuntu 22.04 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-os7tw2atlkwpg) - [NGINX Plus with F5 WAF & DoS for NGINX Plus – Ubuntu 24.04 Linux AMI HVM](https://aws.amazon.com/marketplace/pp/prodview-pz64pqetwyrhw) Click the **Continue to Subscribe** button to proceed to the **Launch on EC2** page. 3. Select the type of launch by clicking the appropriate tab (1‑Click Launch, **Manual Launch**, or **Service Catalog**). Choose the desired options for billing, instance size, and so on, and click the Accept Software Terms… button. 4. When configuring the firewall rules, add a rule to accept web traffic on TCP ports 80 and 443 (this happens automatically if you launch from the 1-Click Launch tab). 5. As soon as the new EC2 instance launches, NGINX Plus starts automatically and serves a default **index.html** page. To view the page, use a web browser to access the public DNS name of the new instance. You can also check the status of the NGINX Plus server by logging into the EC2 instance and running this command: ```nginx /etc/init.d/nginx status ``` See [NGINX Plus on the AWS Cloud](https://www.nginx.com/resources/datasheets/nginx-quick-start-guide-for-aws/) deployment guide for details. 6. Verify latest NGINX PLUS / F5 DoS for NGINX / F5 WAF for NGINX packages are installed on EC2 after its first start: Verify F5 WAF for NGINX latest release from is installed by comparing with installed version from following command on the EC2 machine ```shell cat /opt/app_protect/VERSION /opt/app_protect/RELEASE ``` Verify F5 DoS for NGINX latest release from is installed by comparing with installed version from following command on the EC2 machine ```shell admd -v ``` Verify NGINX Plus latest release from is installed by comparing with installed version from following command on the EC2 machine ```shell nginx -v ``` In case NGINX PLUS / F5 DoS for NGINX / F5 WAF for NGINX packages are not latest release then upgrade the following with these commands: For F5 DoS for NGINX solution based on RedHat / AmazonLinux 2023 ```shell sudo service nginx stop sudo yum install app-protect-dos sudo systemctl start nginx ``` For F5 DoS for NGINX solution based on Debian / Ubuntu ```shell sudo service nginx stop sudo apt-get update sudo apt-get install app-protect-dos sudo service nginx start ``` For F5 WAF for NGINX solution based on RedHat / AmazonLinux 2023 ```shell sudo service nginx stop sudo yum install app-protect sudo systemctl start nginx ``` For F5 WAF for NGINX solution based on Debian / Ubuntu ```shell sudo service nginx stop sudo apt-get update sudo apt-get install app-protect sudo service nginx start ``` 7. If AMI includes [F5 WAF for NGINX](https://docs.nginx.com/nginx-app-protect-waf/) To enable F5 WAF for NGINX use the following steps: a. Load the F5 WAF for NGINX module on the main context in the `nginx.conf` file: ```shell load_module modules/ngx_http_app_protect_module.so; ``` b. Enable F5 WAF for NGINX on an `http/server/location` context in the `nginx.conf` file: ```shell app_protect_enable on; ``` c. Restart the NGINX service: ```shell sudo systemctl restart nginx ``` For more configuration information follow [F5 WAF for NGINX Configuration Guide](https://docs.nginx.com/nginx-app-protect-waf/configuration-guide/configuration/). 8. If AMI includes [F5 DoS for NGINX](https://docs.nginx.com/nginx-app-protect-dos/) To enable F5 DoS for NGINX use the following steps: a. Load the F5 DoS for NGINX module on the main context in the `nginx.conf` file: ```shell load_module modules/ngx_http_app_protect_dos_module.so; ``` b. Enable F5 DoS for NGINX on an `http/server/location` context in the `nginx.conf` file: ```shell app_protect_dos_enable on; app_protect_dos_name "App1"; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; app_protect_dos_monitor uri=serv:80/; # Assuming server_name "serv" on port 80, with the root path "/" ``` c. Enable the L4 accelerated mitigation feature in the `http` context of the `nginx.conf` file: ```shell app_protect_dos_accelerated_mitigation on; ``` d. Restart the NGINX service: ```shell sudo systemctl restart nginx ``` For more configuration information follow [F5 DoS for NGINX Directives and Policy](https://docs.nginx.com/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/). ## What If I Need Help? If you encounter any problems with NGINX Plus configuration, documentation is available at [nginx.org](https://nginx.org/en/docs/) and in the [NGINX Plus Admin Guide](https://docs.nginx.com/nginx/admin-guide/installing-nginx/). If you encounter any problems with F5 DoS for NGINX configuration, documentation is available at the [F5 DoS for NGINX Troubleshooting Guide](https://docs.nginx.com/nginx-app-protect-dos/troubleshooting/how-to-troubleshoot/). If you encounter any problems with F5 WAF for NGINX configuration, documentation is available at the [F5 WAF for NGINX Troubleshooting Guide](https://docs.nginx.com/nginx-app-protect-waf/). Customers who purchase an NGINX Plus AMI at the AWS Marketplace are eligible for the AWS support provided by the NGINX, Inc. engineering team. To activate support, submit the [AMI Support Activation](https://www.nginx.com/ami-support-activation/) form (you need your AWS account number). When you request support, we’ll ask you to provide the AWS account number that you registered, along with the IDs of your EC2 instances in some cases. -------------------------------------------------------------------------------- title: "Kubernetes with L4 accelerated mitigation" url: https://docs.nginx.com/nginx-app-protect-dos/deployment-guide/kubernetes-with-l4-accelerated-mitigation/index.md description: Install F5 DoS for NGINX on Kubernetes with L4 accelerated mitigation using eBPF to offload DoS blocking to the Linux kernel. -------------------------------------------------------------------------------- This guide explains how to install F5 DoS for NGINX on Kubernetes with L4 accelerated mitigation. By enabling the [`app_protect_dos_accelerated_mitigation`](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#accelerated-mitigation-directive-app_protect_dos_accelerated_mitigation) directive and running the DoS eBPF (Extended Berkeley Packet Filter) Manager as a sidecar container alongside the NGINX container, you can offload Layer 4 DoS mitigation to eBPF programs in the Linux kernel. This improves mitigation performance and reduces CPU usage on the NGINX container. Deployments with L4 accelerated mitigation require the NGINX and DoS containers to run with elevated privileges and additional Linux capabilities. This guide assumes you have a good understanding of Kubernetes security best practices and have secured your cluster accordingly. F5 DoS for NGINX requires the service to run with [`externalTrafficPolicy`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) set to `Local` to preserve the client source IP address for accurate DoS mitigation: ```yaml spec: externalTrafficPolicy: Local ``` It covers the common steps for any Kubernetes-based deployment, then provides the manifest-based installation steps. ## Before you begin Before you start, make sure you have: - A functional Kubernetes cluster - An active F5 DoS for NGINX subscription (purchased or trial) - [Docker](https://docs.docker.com/get-started/get-docker/) To review supported operating systems, read the [Releases](/nap-dos/releases) topic. ## Download your subscription credentials Download the SSL certificate, private key, and the JWT license file associated with your NGINX Plus subscription from the MyF5 Customer Portal: - Log in to [MyF5](https://my.f5.com/manage/s/). - Go to **My Products & Plans > Subscriptions** to see your active subscriptions. - Find your NGINX products or services subscription, and select the **Subscription ID** for details. - Download the **.crt** and **.key** from the subscription page. - Download the **JSON Web Token** (JWT) from the subscription page. See [About Subscription Licenses](/solutions/about-subscription-licenses.md) for details. ## Create a Dockerfile In the same folder as your credential files, create a _Dockerfile_ based on your desired operating system image using an example from the following sections. As written, the examples install the most recent version of F5 DoS for NGINX and NGINX Plus. To build a specific version instead, set the `DOS_VERSION` build argument, using the syntax shown in the comment at the top of the Dockerfile for your operating system. On Debian and Ubuntu, also set `NGINX_PLUS_VERSION`, because `apt` considers only the newest `nginx-plus` available and does not select an older one to satisfy the module's dependency. To review which NGINX Plus release each version requires, read the [Releases](/nap-dos/releases) topic. **Note:** Build the NGINX Plus image and the eBPF Manager image from the same F5 DoS for NGINX version. They are built separately, so if you pin one and not the other, the eBPF Manager sidecar can end up running a different version to the NGINX container. ### Alpine Linux #### NGINX Plus ```dockerfile # syntax=docker/dockerfile:1 # Supported OS_VER's are 3.21/3.22 ARG OS_VER="3.22" # Base image FROM alpine:${OS_VER} # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="=37+4.9.6-r1" # NGINX Plus needs no pin here, because apk can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/apk/cert.pem,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/apk/cert.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub \ && printf "https://pkgs.nginx.com/plus/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories \ && printf "https://pkgs.nginx.com/app-protect-dos/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories \ && apk update \ && apk add "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log \ && rm -rf /var/cache/apk/* # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # syntax=docker/dockerfile:1 # Supported OS_VER's are 3.21/3.22 ARG OS_VER="3.22" # Base image FROM alpine:${OS_VER} # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="=37+4.9.6-r1" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/apk/cert.pem,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/apk/cert.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && addgroup -S -g 101 nginx \ && adduser -S -u 101 -G nginx -h /nonexistent -s /sbin/nologin nginx \ && wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub \ && printf "https://pkgs.nginx.com/app-protect-dos/alpine/v`egrep -o '^[0-9]+\.[0-9]+' /etc/alpine-release`/main\n" | tee -a /etc/apk/repositories \ && apk update \ && apk add "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && rm -rf /var/cache/apk/* STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### Amazon Linux #### NGINX Plus ```dockerfile # For AmazonLinux 2023: FROM amazonlinux:2023 # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-amazonlinux2023.repo https://cs.nginx.com/static/files/plus-amazonlinux2023.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo https://cs.nginx.com/static/files/app-protect-dos-amazonlinux2023.repo \ && dnf install -y "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && dnf clean all \ && rm -rf /var/cache/dnf \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # For AmazonLinux 2023: FROM amazonlinux:2023 # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="-37+4.9.6" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ && dnf -y install ca-certificates shadow-utils \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && curl -o /etc/yum.repos.d/app-protect-dos-amazonlinux2023.repo https://cs.nginx.com/static/files/app-protect-dos-amazonlinux2023.repo \ && dnf install -y "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && dnf clean all \ && rm -rf /var/cache/dnf STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### Debian #### NGINX Plus ```dockerfile # Where can be bullseye/bookworm/trixie FROM debian:bullseye # Leave both arguments empty to install the most recent version of each package. # To build a specific version, set both, for example: # --build-arg DOS_VERSION="=37+4.9.6-1~bullseye" --build-arg NGINX_PLUS_VERSION="=37.0.*-1~bullseye" # Pin NGINX Plus as well: apt considers only the newest nginx-plus available and # does not select an older one to satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" ARG NGINX_PLUS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update \ && DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 debian-archive-keyring \ && mkdir -p /etc/ssl/nginx/ /etc/nginx/ \ && wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/debian $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/debian $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \ && apt-get update \ && DEBIAN_FRONTEND="noninteractive" apt-get install -y "app-protect-dos${DOS_VERSION}" "nginx-plus-module-appprotectdos${DOS_VERSION}" "nginx-plus${NGINX_PLUS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && apt-get remove --purge --auto-remove -y apt-transport-https lsb-release gnupg2 wget \ && rm -rf /var/lib/apt/lists/* \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # Where can be bullseye/bookworm FROM debian:bullseye # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="=37+4.9.6-1~bullseye" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && DEBIAN_FRONTEND=noninteractive apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ apt-transport-https \ lsb-release \ ca-certificates \ wget \ gnupg2 \ debian-archive-keyring \ && wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key \ | gpg --dearmor \ | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null \ && echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/debian $(lsb_release -cs) nginx-plus" \ > /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \ && DEBIAN_FRONTEND=noninteractive apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### RHEL 8 #### NGINX Plus ```dockerfile # For UBI 8 FROM registry.access.redhat.com/ubi8 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} \ && subscription-manager refresh \ && subscription-manager attach --auto || true \ && subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms \ && subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-8.repo https://cs.nginx.com/static/files/plus-8.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-8.repo https://cs.nginx.com/static/files/app-protect-dos-8.repo \ && dnf -y install "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && rm /etc/yum.repos.d/plus-8.repo \ && rm /etc/yum.repos.d/app-protect-dos-8.repo \ && dnf clean all \ && rm -rf /var/cache/yum \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # For UBI 8 FROM registry.access.redhat.com/ubi8 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="-37+4.9.6" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/app-protect-dos-8.repo https://cs.nginx.com/static/files/app-protect-dos-8.repo \ && dnf -y install "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && rm /etc/yum.repos.d/app-protect-dos-8.repo \ && dnf clean all \ && rm -rf /var/cache/yum STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### RHEL 9 #### NGINX Plus ```dockerfile # For UBI 9 FROM registry.access.redhat.com/ubi9 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} \ && subscription-manager refresh \ && subscription-manager attach --auto || true \ && subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms \ && subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-9.repo https://cs.nginx.com/static/files/plus-9.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo \ && dnf -y install "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && rm /etc/yum.repos.d/plus-9.repo \ && rm /etc/yum.repos.d/app-protect-dos-9.repo \ && dnf clean all \ && rm -rf /var/cache/yum \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # For UBI 9 FROM registry.access.redhat.com/ubi9 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="-37+4.9.6" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo \ && dnf -y install "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && rm /etc/yum.repos.d/app-protect-dos-9.repo \ && dnf clean all \ && rm -rf /var/cache/yum STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### RHEL 10 #### NGINX Plus ```dockerfile # For UBI 10 FROM registry.access.redhat.com/ubi10 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ subscription-manager register --org=${RHEL_ORG} --activationkey=${RHEL_ACTIVATION_KEY} \ && subscription-manager refresh \ && subscription-manager attach --auto || true \ && subscription-manager repos --enable=rhel-10-for-x86_64-baseos-rpms \ && subscription-manager repos --enable=rhel-10-for-x86_64-appstream-rpms \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/plus-10.repo https://cs.nginx.com/static/files/plus-10.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-10.repo https://cs.nginx.com/static/files/app-protect-dos-10.repo \ && dnf -y install "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && rm /etc/yum.repos.d/plus-10.repo \ && rm /etc/yum.repos.d/app-protect-dos-10.repo \ && dnf clean all \ && rm -rf /var/cache/yum \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # For UBI 10 FROM registry.access.redhat.com/ubi10 ARG RHEL_ORG ARG RHEL_ACTIVATION_KEY # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="-37+4.9.6" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm \ && dnf -y install ca-certificates \ && curl -o /etc/yum.repos.d/app-protect-dos-10.repo https://cs.nginx.com/static/files/app-protect-dos-10.repo \ && dnf -y install "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && rm /etc/yum.repos.d/app-protect-dos-10.repo \ && dnf clean all \ && rm -rf /var/cache/yum STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### Rocky Linux 9 #### NGINX Plus ```dockerfile # syntax=docker/dockerfile:1 # For Rocky Linux 9 FROM rockylinux:9 # Leave empty to install the most recent version. To build a specific version, # set the argument, for example: # --build-arg DOS_VERSION="-37+4.9.6" # NGINX Plus needs no pin here, because dnf can select an older nginx-plus to # satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" # Install F5 DoS for NGINX: RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ dnf -y install ca-certificates epel-release 'dnf-command(config-manager)' \ && curl -o /etc/yum.repos.d/plus-9.repo https://cs.nginx.com/static/files/plus-9.repo \ && curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo \ && dnf config-manager --set-enabled crb \ && dnf install -y "app-protect-dos${DOS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && dnf clean all \ && rm -rf /var/cache/dnf \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log # Copy configuration files: COPY nginx.conf custom_log_format.json /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # syntax=docker/dockerfile:1 # For Rocky Linux 9 FROM rockylinux:9 # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="-37+4.9.6" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && dnf -y install ca-certificates epel-release 'dnf-command(config-manager)' \ && curl -o /etc/yum.repos.d/app-protect-dos-9.repo https://cs.nginx.com/static/files/app-protect-dos-9.repo \ && dnf config-manager --set-enabled crb \ && dnf install -y "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && dnf clean all \ && rm -rf /var/cache/dnf STOPSIGNAL SIGQUIT CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ### Ubuntu #### NGINX Plus ```dockerfile # syntax=docker/dockerfile:1 # For Ubuntu # Where version can be: jammy/noble FROM ubuntu:noble # Leave both arguments empty to install the most recent version of each package. # To build a specific version, set both, for example: # --build-arg DOS_VERSION="=37+4.9.6-1~noble" --build-arg NGINX_PLUS_VERSION="=37.0.*-1~noble" # Pin NGINX Plus as well: apt considers only the newest nginx-plus available and # does not select an older one to satisfy the module's nginx-plus-r dependency. ARG DOS_VERSION="" ARG NGINX_PLUS_VERSION="" # Install F5 DoS for NGINX RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ --mount=type=secret,id=license-jwt,dst=license.jwt,mode=0644 \ apt-get update \ && apt-get install -y --no-install-recommends apt-transport-https lsb-release ca-certificates wget gnupg2 ubuntu-keyring \ && mkdir -p /etc/ssl/nginx/ /etc/nginx/ \ && wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/plus/ubuntu $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-plus.list \ && printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/ubuntu $(lsb_release -cs) nginx-plus\n" > /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \ && apt-get update \ && DEBIAN_FRONTEND="noninteractive" apt-get install -y "app-protect-dos${DOS_VERSION}" "nginx-plus-module-appprotectdos${DOS_VERSION}" "nginx-plus${NGINX_PLUS_VERSION}" \ && cat license.jwt > /etc/nginx/license.jwt \ && apt-get remove --purge --auto-remove -y apt-transport-https lsb-release gnupg2 wget && rm -rf /var/lib/apt/lists/* \ && ln -sf /dev/stdout /var/log/nginx/access.log \ && ln -sf /dev/stderr /var/log/nginx/error.log COPY nginx.conf /etc/nginx/ COPY entrypoint.sh /root/ RUN chmod +x /root/entrypoint.sh EXPOSE 80 STOPSIGNAL SIGQUIT CMD ["sh", "/root/entrypoint.sh"] ``` #### EBPF Manager ```dockerfile # syntax=docker/dockerfile:1 # For Ubuntu # Where version can be: jammy/noble FROM ubuntu:noble # Set this to the same version you built the F5 DoS for NGINX image with, for example: # --build-arg DOS_VERSION="=37+4.9.6-1~noble" # Left empty, the most recent version is installed, which may not match that image. ARG DOS_VERSION="" # Install F5 DoS ebpf manager for NGINX and create required nginx user RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \ --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \ set -x \ # Create nginx user/group first, to be consistent throughout Docker variants \ && groupadd --system --gid 101 nginx \ && useradd --system --gid nginx --no-create-home --home /nonexistent --comment "nginx user" --shell /bin/false --uid 101 nginx \ && DEBIAN_FRONTEND=noninteractive apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ apt-transport-https \ lsb-release \ ca-certificates \ wget \ gnupg2 \ ubuntu-keyring \ && wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key \ | gpg --dearmor \ | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null \ && echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://pkgs.nginx.com/app-protect-dos/ubuntu $(lsb_release -cs) nginx-plus" \ > /etc/apt/sources.list.d/nginx-app-protect-dos.list \ && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \ && DEBIAN_FRONTEND=noninteractive apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y "app-protect-dos-ebpf-manager${DOS_VERSION}" \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* STOPSIGNAL SIGQUIT # Idle forever CMD ["bash", "-c", "/usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log"] ``` ## Create DOS entrypoint.sh Docker startup script which spins up all App Protect DoS processes, must have executable permissions ```shell #!/usr/bin/env bash USER=nginx LOGDIR=/var/log/adm # prepare environment mkdir -p /var/run/adm /tmp/cores ${LOGDIR} chmod 755 /var/run/adm /tmp/cores ${LOGDIR} chown ${USER}:${USER} /var/run/adm /tmp/cores ${LOGDIR} # run processes /bin/su -s /bin/bash -c "/usr/bin/adminstall > ${LOGDIR}/adminstall.log 2>&1" ${USER} /bin/su -s /bin/bash -c "/usr/bin/admd -d --log info > ${LOGDIR}/admd.log 2>&1 &" ${USER} /usr/sbin/nginx -g 'daemon off;' ``` ## Build the DOS Docker image Your folder should contain the following files: - _nginx-repo.crt_ - _nginx-repo.key_ - _license.jwt_ - _entrypoint.sh_ - _nginx.conf_ - _Dockerfile_ To build an image, use the following command, replacing `` as appropriate: ```shell sudo docker build --no-cache --platform linux/amd64 \ --secret id=nginx-crt,src=nginx-repo.crt \ --secret id=nginx-key,src=nginx-repo.key \ -t . ``` To build a specific version instead, add the two build arguments. For example, on Debian 11: ```shell sudo docker build --no-cache --platform linux/amd64 \ --secret id=nginx-crt,src=nginx-repo.crt \ --secret id=nginx-key,src=nginx-repo.key \ --build-arg DOS_VERSION="=37+4.9.6-1~bullseye" \ --build-arg NGINX_PLUS_VERSION="=37.0.*-1~bullseye" \ -t . ``` ## Build the EBPF Manager Docker image Your folder should contain the following files: - _nginx-repo.crt_ - _nginx-repo.key_ - _Dockerfile_ To build an image, use the following command, replacing `` as appropriate: ```shell sudo docker build --no-cache --platform linux/amd64 \ --secret id=nginx-crt,src=nginx-repo.crt \ --secret id=nginx-key,src=nginx-repo.key \ -t . ``` If you pinned a version when building the NGINX Plus image, pass the same `DOS_VERSION` here: ```shell sudo docker build --no-cache --platform linux/amd64 \ --secret id=nginx-crt,src=nginx-repo.crt \ --secret id=nginx-key,src=nginx-repo.key \ --build-arg DOS_VERSION="=37+4.9.6-1~bullseye" \ -t . ``` Once you have built the DOS and EBPF images, push them to your private image repository, which should be accessible to your Kubernetes cluster. ## Use Manifests to install F5 DOS for NGINX ### Create Manifest files The default configuration provided creates two replicas, each hosting NGINX and DOS services together in a single Kubernetes pod. Create all of these files in a single folder (Such as `/manifests`). On manifest deployment environment variables need to be set for image repository and tag. `set enviorment variable DOS_IMAGE_REPOSITORY` with your actual nginx-dos image anmae. `set enviorment variable DOS_IMAGE_TAG` with your actual nginx-dos image tag. `set enviorment variable EBPF_IMAGE_REPOSITORY` with your actual ebpf-manager image name. `set enviorment variable EBPF_IMAGE_TAG` with your actual ebpf-manager image tag. ```dos-namespace.yaml apiVersion: v1 kind: Namespace metadata: name: app-protect-dos ``` ```dos-nginx-conf-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: dos-nginx-conf namespace: app-protect-dos data: nginx.conf: | user nginx; worker_processes auto; error_log /var/log/nginx/error.log error; worker_rlimit_nofile 65535; load_module modules/ngx_http_app_protect_dos_module.so; working_directory /tmp/cores; events { worker_connections 65535; } http { app_protect_dos_arb_fqdn svc-appprotect-dos-arb.arb.svc.cluster.local; sendfile on; tcp_nopush on; keepalive_timeout 65; log_format log_dos ', vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, ' 'outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, ' 'ip_tls=$remote_addr:$app_protect_dos_tls_fp, '; app_protect_dos_accelerated_mitigation on syn_drop=on; # Health endpoints for probes app_protect_dos_liveness on; # uri:/app_protect_dos_liveness port:8090 app_protect_dos_readiness on; # uri:/app_protect_dos_readiness port:8090 server { listen 8090; server_name probe; location / { proxy_pass http://localhost:8091; } } server { listen 8091; return 503; } server { listen 80 reuseport; server_name serv; access_log /var/log/nginx/access.log log_dos if=$loggable; app_protect_dos_security_log_enable on; app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=:5261; app_protect_dos_policy_file "/etc/app_protect_dos/BADOSDefaultPolicy.json"; location / { app_protect_dos_enable on; app_protect_dos_name "main_serv"; app_protect_dos_monitor uri=http://serv:80/ protocol=http1; proxy_pass http://127.0.0.1/proxy$request_uri; } location /proxy { app_protect_dos_enable off; client_max_body_size 0; default_type text/html; return 200 "Hello! I got your URI request - $request_uri\n"; } } } ``` ```dos-log-default-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: dos-log-default namespace: app-protect-dos data: log-default.json: | { "filter": { "traffic-mitigation-stats": "all", "bad-actors": "all", "attack-signatures": "all" } } ``` ```dos-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: app-protect-dos namespace: app-protect-dos labels: app: app-protect-dos spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: app-protect-dos strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 template: metadata: labels: app: app-protect-dos spec: containers: - name: nginx-app-protect-dos image: ${DOS_IMAGE_REPOSITORY}:${DOS_IMAGE_TAG} imagePullPolicy: Always command: ["/bin/bash", "-c"] args: - | /root/entrypoint.sh resources: requests: cpu: "200m" memory: "500Mi" limits: cpu: "900m" memory: "800Mi" ports: - containerPort: 80 name: web - containerPort: 8090 name: probe - containerPort: 8091 name: probe500 livenessProbe: httpGet: path: /app_protect_dos_liveness port: 8090 initialDelaySeconds: 5 periodSeconds: 10 readinessProbe: httpGet: path: /app_protect_dos_readiness port: 8090 initialDelaySeconds: 5 periodSeconds: 10 volumeMounts: - name: shared-dir mountPath: /shared/ - name: bpf mountPath: /sys/fs/bpf - name: conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf - name: log-default mountPath: /etc/app_protect_dos/log-default.json subPath: log-default.json - name: license-token-volume mountPath: /etc/nginx/license.jwt subPath: license.jwt readOnly: true - name: dos-ebpf-manager image: ${EBPF_IMAGE_REPOSITORY}:${EBPF_IMAGE_TAG} securityContext: privileged: true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name # This is unique for every Pod volumeMounts: - name: shared-dir mountPath: /shared/ - name: bpf mountPath: /sys/fs/bpf volumes: - name: shared-dir emptyDir: {} - name: bpf hostPath: path: /sys/fs/bpf type: DirectoryOrCreate - name: conf configMap: name: dos-nginx-conf items: - key: nginx.conf path: nginx.conf - name: log-default configMap: name: dos-log-default defaultMode: 0644 items: - key: log-default.json path: log-default.json - name: license-token-volume secret: secretName: license-token items: - key: license.jwt path: license.jwt ``` ```dos-service.yaml apiVersion: v1 kind: Service metadata: name: nap-dos namespace: app-protect-dos spec: externalTrafficPolicy: Local ports: - name: app port: 80 protocol: TCP selector: app: app-protect-dos type: LoadBalancer ``` ### Start the Manifest deployment From the folder containing the YAML files from the previous step (Suggested as `/manifests`), deploy F5 DOS for NGINX using `kubectl`: ```shell export DOS_IMAGE_REPOSITORY= export DOS_IMAGE_TAG= export EBPF_IMAGE_REPOSITORY= export EBPF_IMAGE_TAG= kubectl apply -f manifests/dos-namespace.yaml kubectl apply -f manifests/dos-nginx-conf-configmap.yaml kubectl apply -f manifests/dos-log-default-configmap.yaml kubectl apply -f manifests/dos-deployment.yaml kubectl apply -f manifests/dos-service.yaml ``` It will apply all the configuration defined in the files to your Kubernetes cluster. You can then check the status of the deployment with `kubectl get`: ```shell kubectl -n app-protect-dos get deployments kubectl -n app-protect-dos get pods kubectl -n app-protect-dos get services ``` You should see output similar to the following: ```text ~$ kubectl -n app-protect-dos get deployments NAME READY UP-TO-DATE AVAILABLE AGE app-protect-dos 1/1 1 1 33s ~$ kubectl -n app-protect-dos get pods NAME READY STATUS RESTARTS AGE app-protect-dos-7f9798654c-7ncbl 2/2 Running 0 68s $ kubectl -n app-protect-dos get pods -o jsonpath='{range .items[*]}Pod: {.metadata.name} -> Containers: {.spec.containers[*].name}{"\n"}{end}' Pod: app-protect-dos-7f9798654c-7ncbl -> Containers: dos-ebpf-manager nginx-app-protect-dos ~$ kubectl -n app-protect-dos get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nap-dos LoadBalancer 10.43.212.232 80:32586/TCP 93s ``` ## Post-Installation Checks At this stage, you have finished deploying F5 DOS for NGINX with EBPF L4 accelerated mitigation enabled You can login to dos-ebpf-manager container like following command ```text kubectl exec -it app-protect-dos-586fb94947-8sjnc -n app-protect-dos -c nginx-app-protect-dos -- bash kubectl exec -it app-protect-dos-586fb94947-8sjnc -n app-protect-dos -c dos-ebpf-manager -- bash ``` and can look at . You can run the following commands to ensure that F5 DoS for NGINX enforcement is operational. 1. Check that the three processes needed for F5 DoS for NGINX are running using `ps aux`: - admd - nginx: master process - nginx: worker process ```shell USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND nginx 7759 0.0 0.0 113120 1200 ? Ss Sep06 0:00 /bin/sh -c /usr/bin/admd -d --log info > /var/log/adm/admd.log 2>&1 root 7765 0.0 0.0 87964 1464 ? Ss Sep06 0:00 nginx: master process /usr/sbin/nginx -g daemon off; nginx 7767 0.0 0.1 615868 8188 ? Sl Sep06 0:04 nginx: worker process ``` 2. Verify that there are no NGINX errors in the `/var/log/nginx/error.log` and that the policy compiled successfully: ```shell 2025/12/07 09:14:34 [notice] 675#675: APP_PROTECT_DOS { "event": "shared_memory_connected", "worker_pid": 675, "mode": "operational", "mode_changed": true } 2025/12/07 09:14:34 [notice] 675#675: using the "epoll" event method 2025/12/07 09:14:34 [notice] 675#675: APP_PROTECT_DOS { "event": "configuration_load_success", "software_version": "36+4.8.3-1.el8.ngx"} 2025/12/07 09:14:34 [notice] 675#675: nginx/1.29.3 (nginx-plus-r36) 2025/12/07 09:14:34 [notice] 675#675: built by gcc 8.5.0 20210514 (Red Hat 8.5.0-28) (GCC) 2025/12/07 09:14:34 [notice] 675#675: OS: Linux 6.8.0-88-generic 2025/12/07 09:14:34 [notice] 675#675: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2025/12/07 09:14:34 [notice] 675#675: start worker processes 2025/12/07 09:14:34 [notice] 675#675: start worker process 679 2025/12/07 09:14:34 [notice] 679#679: APP_PROTECT_DOS { "event": "shared_memory_connected", "worker_pid": 679, "mode": "operational", "mode_changed": true } ``` 3. Verify that when you simulate an attack, attacker IP addresses are blocked while legitimate traffic passes through: a. Simulate good traffic: ```shell echo "Start Good Traffic 2" while true; do curl ${VS}/good1 & curl ${VS}/good2 & curl ${VS}/good3 & curl ${VS}/good4 sleep 0.1 done & ``` b. After 7 minutes start the attack: ```shell while [ true ] do ab -B ${BAD_IP1} -l -r -n 1000000 -c 150 -d -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ & ab -B ${BAD_IP2} -l -r -n 1000000 -c 150 -d -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ & ab -B ${BAD_IP3} -l -r -n 1000000 -c 150 -d -s 10 -H "Host: evil.net" -H "Pragma: no-cache" -H "Cache-Control: no-cache" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: WireXBot" -H "x-requested-with:" -H "Referer: http://10.0.2.1/none.html" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US" http://${VS}/ killall ab done ``` c. Verify that legitimate traffic continues as usual while the attack traffic is blocked. 4. For DOS with L4 accelerated mitigation enabled Check that the ebpf_manager_dos process needed for F5 DoS for NGINX is running using `ps aux | grep /usr/bin/ebpf_manager_dos`: ``` root 1 0.0 0.0 4324 3072 ? Ss 19:32 0:00 bash -c /usr/bin/ebpf_manager_dos 2>&1 | tee /shared/ebpf_dos.log root 7 0.2 0.0 1722732 14208 ? Sl 19:32 0:01 /usr/bin/ebpf_manager_dos root 46 0.0 0.0 3528 1792 pts/0 S+ 19:44 0:00 grep --color=auto /usr/bin/ebpf_manager_dos ``` Verify that there are no errors in the `/shared/ebpf_dos.log` and that the XDP program uploaded successfully: ```[2025-12-02 19:32:12] INFO: Uninstall old eBPF maps and XDP program [2025-12-02 19:32:13] INFO: Install eBPF maps and XDP program [2025-12-02 19:32:13] INFO: Start ebpf manager [2025-12-02 19:32:13] INFO: Version: 36+4.8.3-1~noble [2025-12-02 19:32:13] INFO: Start Periodic task for update time [2025-12-02 19:32:13] INFO: Owner of the UDS has been changed to user nginx and group nginx. [2025-12-02 19:32:13] INFO: Permissions of the UDS have been changed successfully for user nginx and group nginx. [2025-12-02 19:32:13] INFO: Async Callback Server listening on unix:/shared/ebpf_manager_dos_uds ``` To check F5 WAF for NGINX alongside F5 DoS for NGINX, just perform the normal tests as specified at [Admin Guide](https://docs.nginx.com/waf/install/virtual-environment/#post-installation-checks) ## F5 DoS for NGINX Arbitrator ## F5 DoS for NGINX Arbitrator ### Overview F5 DoS for NGINX Arbitrator orchestrates all running F5 DoS for NGINX instances to synchronize local and global attack start and stop. F5 DoS for NGINX Arbitrator is a central coordinating component for managing multiple F5 DoS for NGINX instances in a network. It is needed when there is more than one F5 DoS for NGINX instance. Its primary function is to ensure that all instances are aware of and share the same state for each protected object. ### How the Arbitrator works - **Collecting state periodically**: The Arbitrator regularly collects state information from all running F5 DoS for NGINX instances. This collection occurs at set intervals, typically every 10 seconds. - **State initialization for new instances**: When a new F5 DoS for NGINX instance starts, it retrieves the initial state for each protected object from the Arbitrator rather than starting with an empty state. - **Updating state during an attack**: When an F5 DoS for NGINX instance detects an attack, it sends a notification to the Arbitrator. The Arbitrator updates the state of the affected protected object and propagates that state to all other instances. ### Why F5 DoS for NGINX Arbitrator is necessary F5 DoS for NGINX Arbitrator is essential for several reasons: - **Global state management**: Without the Arbitrator, each F5 DoS for NGINX instance manages its own isolated state for each protected object. This can lead to inconsistencies. For example, if instance A declares an attack on a protected object named "PO-Example," instance B remains unaware of it, potentially leaving the object vulnerable. - **Uniform attack detection**: With the Arbitrator, when instance A detects an attack on "PO-Example" and reports it, the Arbitrator updates the state of "PO-Example" and propagates it to all instances, including instance B. F5 DoS for NGINX Arbitrator maintains a consistent global state for protected objects across all F5 DoS for NGINX instances. This ensures attacks are detected and mitigated uniformly across your deployment. ### F5 DoS for NGINX Arbitrator Deployment 1. Pull the official F5 DoS for NGINX Arbitrator image with the command: ```shell docker pull docker-registry.nginx.com/nap-dos/app_protect_dos_arb:latest ``` 2. Create a container based on this image, for example, `app-protect-dos-arb` container: ```shell docker run --name app_protect_dos_arb -p 3000:3000 -d docker-registry.nginx.com/nap-dos/app_protect_dos_arb ``` 3. Verify that the `app-protect-dos-arb` container is up and running with the `docker ps` command. 4. DNS records are required for F5 DoS for NGINX Arbitrator to work properly and be accessible by F5 DoS for NGINX servers. Ensure that the `svc-appprotect-dos-arb` or configured Arbitrator FQDN (with `app_protect_dos_arb_fqdn` directive) has a valid DNS resolution. This step is necessary only for VM/Docker deployments with arbitrator. When the arbitrator is in the same Kubernetes namespace as F5 DoS for NGINX, this step is not needed. ### Multi-VM Deployment The Arbitrator service is standalone. If it goes down, it can be restarted and immediately recovers all required information from F5 DoS for NGINX instances, which report to it every 10 seconds. Its downtime is around 10 to 20 seconds, which does not affect F5 DoS for NGINX operation. F5 DoS for NGINX Arbitrator connects to port 3000. All modules try to connect to it automatically. If it's not accessible, each instance operates in standalone mode. F5 DoS for NGINX does not support mutual TLS (mTLS) or password authentication between DoS servers and the Arbitrator. Arbitrator is not exposed outside the namespace. It is the customer's responsibility to isolate it from external access. This applies to all Arbitrator deployments, not only multi-VM. ## Next steps -------------------------------------------------------------------------------- title: "F5 DoS for NGINX Arbitrator 1.1.1" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-1.1.1/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX Arbitrator v1.1.1. ## Arbitrator Service Release 1.1.1 May 2, 2023 This release is focused on security. New image is backward compatible with the previous version. ### Resolved Issues - Removed unused REDIS module -------------------------------------------------------------------------------- title: "Directives and Policy" url: https://docs.nginx.com/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/index.md -------------------------------------------------------------------------------- ## Introduction NGINX directives are specified in the `nginx.conf` file and are used to configure various modules of NGINX.
F5 DoS for NGINX has its own set of directives, which follow the same rules as other NGINX directives, and are used to enable and configure its features.
The table below provides a summary of all the F5 DoS for NGINX directives.
While only the first directive is mandatory for enabling F5 DoS for NGINX, it is recommended to use as many directives as possible to leverage the product’s full range of monitoring and application health detection capabilities. After adding these directives, ensure you reload NGINX and check the error log for any errors or warnings.
## Directives table Below is a summary of all F5 DoS for NGINX directives. Detailed descriptions of each directive can be found in the following sections. | Directive syntax | Options | Context | Description | Mandatory | Default | |-----------------------------------------------------------------------------------------------------------|----------|----------|--------------|------------|-------------------------------------------------------------------------------------------------------------------------------------| | [app_protect_dos_enable](#enable-directive-app_protect_dos_enable) | [on\|off] | http,
server,
location | Enable/Disable DoS protection | Yes | off | | [app_protect_dos_policy_file](#policy-directive-app_protect_dos_policy_file) | [FILE-PATH] | http,
server,
location | Load DoS configuration from a policy file | No | `/etc/app_protect_dos/BADOSDefaultPolicy.json` | | [app_protect_dos_name](#service-name-directive-app_protect_dos_name) | [SERVICE-NAME] | http,
server,
location | Name of protected object | No | **line_num-server_name**:*seq*-location_name

(i.e. `30-backend:1-/abc`) | | [app_protect_dos_monitor](#monitor-directive-app_protect_dos_monitor) | [uri=X] [protocol=Y] [timeout=Z] [proxy_protocol \| proxy_protocol=on\|off] | http,
server,
location | URI to monitor server's stress. Protocol and timeout are optional | Yes, unless its regular http1 traffic | uri - None
protocol - http1
timeout - 10 seconds for http1/websocket ; 5 seconds for http2/grpc
proxy_protocol - off | | [app_protect_dos_security_log_enable](#security-log-enable-directive-app_protect_dos_security_log_enable) | [on\|off] | http,
server,
location | Enable/Disable security logger | No | off | | [app_protect_dos_security_log](#security-log-directive-app_protect_dos_security_log) | [LOG-CONFIG-FILE] [DESTINATION] | http,
server,
location | Security logger configuration. Second argument:
"syslog:server={ip}:{port}" or
"stderr" or
"{absolute_file_path}" | No | `/etc/app_protect_dos/log-default.json stderr` | | [app_protect_dos_liveness](#liveness-probe-directive-app_protect_dos_liveness) | [on\|off] [uri:URI] [port:PORT] | http | Liveness prob. Second and third arguments are optional | No | `off uri:/app_protect_dos_liveness port:8090` | | [app_protect_dos_readiness](#readiness-probe-directive-app_protect_dos_readiness) | [on\|off] [uri:URI] [port:PORT] | http | Readiness prob. Second and third arguments are optional | No | `off uri:/app_protect_dos_readiness port:8090` | | [app_protect_dos_arb_fqdn](#arbitrator-fqdn-directive-app_protect_dos_arb_fqdn) | [FQDN\|IP address] | http | Arbitrator FQDN/IP address | No | `svc-appprotect-dos-arb` | | [app_protect_dos_api](#api-directive-app_protect_dos_api) | No arguments | location | Monitoring via Rest API (also includes the dashboard) | No | off | | [app_protect_dos_accelerated_mitigation](#api-directive-app_protect_dos_api) | [on\|off] [syn_drop=on\|off]| http | Enable/Disable L4 accelerated mitigation. Second argument is optional | No | off syn_drop=off | | [app_protect_dos_access_file](#access-file-directive-app_protect_dos_access_file) | [FILE-PATH] | http,
server,
location | Define allowlist policy from a file | No | None / disabled | ## Directives Info ### Enable directive (`app_protect_dos_enable`) Enables/disables App Protect DoS module in the relevant block/s.
It can be written in the following contexts: `location/server/http`. The derived blocks/contexts also inherit the directive. **For example:** A directive written in `http` context will be considered as if written also in all of the http's server blocks and their location blocks. In case of multiple directives in different contexts, the derived overwrites the base's directive. | Config | Expected | |------- | -------- | | Http block: directive is **on**
Server block: none is written
Location-1 block: none is written
Location-2 block: none is written | VS1: the server block
VS2: location-1 block
VS3: location-2 block | | Server block: directive is **on**
Location-1 block: directive is **off**
Location-2 block: none is written | VS1: the server block
VS2: location-2 block | | Http block: directive is **on**
Server block: directive is **off**
Location-1 block: directive is **on**
Location-2 block: none is written | VS1: location-1 block | **Example:** ```nginx app_protect_dos_enable on; ``` ### Policy directive (`app_protect_dos_policy_file`) This is the path to the JSON policy file which includes the product's configuration parameters. It can be written in the following contexts: `location/server/http`. The directive is optional. If not inserted then default path will be used, which is `/etc/app_protect_dos/BADOSDefaultPolicy.json`. If the configuration file doesn't exist or its attributes are invalid, default values will be used. `BADOSDefaultPolicy.json`: ```json { "mitigation_mode": "standard", "signatures": "on", "bad_actors": "on", "automation_tools_detection": "on", "tls_fingerprint" : "on" } ``` | Parameter name | Values | Default | Description | |:--------------- |:------- |:--------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | mitigation_mode | standard / conservative / none | standard| **Standard** - module allowed to use global rate mitigation
**Conservative** - module is not allowed to use global rate but only Signatures/Bad Actors mitigation
**None** - module is not allowed to mitigate. Only to learn and report. | | signatures | [on\|off] | on| Enable Signatures mechanism | | bad_actors | [on\|off] | on| Enable Bad Actors mechanism | | automation_tools_detection | [on\|off] | on |Enable the usage of automation tools detection (via cookies and redirect) | | tls_fingerprint| [on\|off] | on | Enable source identification using TLS fingerprinting| | Scenario | Result | |:--------- |:-------- | | Directive is not written | Default path is used: "/etc/app_protect_dos/BADOSDefaultPolicy.json" | | Directive is written | Path from the directive is used | | File not found / file syntax is invalid | Default values are used | **Example:** ```nginx app_protect_dos_policy_file /etc/app_protect_dos/BADOSPolicy.json; ``` ### Service Name directive (`app_protect_dos_name`) This is the Protected Object (VS) name, which should be unique and is used to identify the Protected Object in the logs.
It can be utilized within `location`, `server`, and `http` blocks.

Directive is optional. If not written, then each protected object (VS) will have an auto-generated name according to the following syntax: `line_number-server_name:seq-location_name` **For example:** `30-backend:1-/abc` - `line number:` the line number of the server block (`server {`) in the `nginx.conf` file (i.e. `30`)
- `server name:` taken from directive `server_name` (i.e. `backend`)
seq: 0 for server block, increments for each location block. i.e. VS created from server block will have 0 and VS's from location blocks will be 1,2,3,... (i.e. `1`) - `location name:` the name of the location (i.e. `/abc`) F5 DoS for NGINX supports up to 300 Protected Objects for versions up to 4.3, and 1,000 Protected Objects in version 4.4 and above.

**Example:** ```nginx app_protect_dos_name po-example; ``` ### Monitor directive (`app_protect_dos_monitor`) The `app_protect_dos_monitor` directive is used to monitor the stress level of the Protected Object.
Requests for this monitoring are sent from localhost (127.0.0.1) and pass through the NGINX configuration to simulate regular client traffic.
This directive is mandatory, except when using the `http1` protocol, where it is still strongly recommended for optimal performance.
**Syntax:**
app_protect_dos_monitor uri=path [protocol=http1|http2|grpc|websocket] [timeout=number] [proxy_protocol=on|off]; **Arguments**
Monitor directive has four arguments - **uri**, **protocol**, **timeout** and **proxy_protocol**. The first is mandatory and the rest are optional. - **URI** - The URI of the Protected Object as defined in the `nginx.conf`. This must point to a location block that proxies traffic to the backend (upstream) to ensure accurate monitoring.
Format: **scheme://server_name:port/location**. **Note:** For gRPC, the URI must specify a valid gRPC method (for example, /RouteGuide/GetFeature).
The health check is not a true gRPC client, so its requests do not conform to the gRPC wire protocol. As a result, the backend responds with grpc-status: 12 (UNIMPLEMENTED), which is expected and treated as a successful health check. Regular gRPC client traffic is unaffected by this behavior. - **Protocol** - determines the protocol type of the service. Options are `http1 / http2 / grpc / websocket`.
Default: `http1`.
**Note:** HTTP2 and gRPC are supported from F5 DoS for NGINX v2, while WebSocket is supported from F5 DoS for NGINX v4. - **Timeout** - determines how long (in seconds) should F5 DoS for NGINX wait for a response.
Default: 10 seconds for `http1/http2/websocket` and 5 seconds for `grpc`.
- **Proxy Protocol** - Should be used when the listen directive of the corresponding server block contains the proxy_protocol parameter. It adds an HAProxy PROXY protocol header to the monitor request.
Format is **proxy_protocol | proxy_protocol=on**.
Default: off.
**Note:** The proxy_protocol is supported from F5 DoS for NGINX v3.1. #### For Older Versions (F5 DoS for NGINX v1) In F5 DoS for NGINX v1, the app_protect_dos_monitor directive has only one argument: uri. Only HTTP1 is supported.

**Examples:** 1. HTTP/1 on Port 80: ```nginx listen 80; server_name serv; location / { # Protected Object is defined here app_protect_dos_monitor uri=http://serv:80/; } ``` **Note:** For F5 DoS for NGINX v1, use: app_protect_dos_monitor ; 2. HTTP/2 Over SSL ```nginx listen 443 http2 reuseport ssl; server_name serv; location / { # Protected Object is defined here app_protect_dos_monitor uri=https://serv:443/ protocol=http2 timeout=5; } ``` 3. gRPC Service on Port 50051 ```nginx listen 50051 http2 reuseport; server_name my_grpc; location /routeguide. { # Protected Object is defined here # Note: The URI must include a valid gRPC method (for example, /routeguide.RouteGuide/GetFeature). # The health check will expect a grpc-status of 12 (UNIMPLEMENTED) because it is not a true gRPC client. app_protect_dos_monitor uri=http://my_grpc:50051/routeguide.RouteGuide/GetFeature protocol=grpc; } ``` 4. Server with Proxy Protocol ```nginx listen 443 ssl http2 proxy_protocol; server_name serv; location / { # Protected Object is defined here # Note: Use proxy_protocol=on if the listen directive includes the "proxy_protocol" parameter. app_protect_dos_monitor uri=https://serv:443/ protocol=http2 timeout=5 proxy_protocol=on; } location /abc { # Protected Object is defined here app_protect_dos_monitor uri=https://serv:443/abc protocol=http2 timeout=5 proxy_protocol; } ``` 5. WebSocket service ```nginx listen 80; server_name wsserv; location /app/ { # WebSocket configuration required by NGINX proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; # Protected Object is defined here app_protect_dos_monitor uri=http://wsserv:80/app/ protocol=websocket; } ``` ### Security log enable directive (`app_protect_dos_security_log_enable`) Enable/Disable App Protect DoS security logger. It can be used in `location/server/http` blocks. Directive is optional. If not written, then logger is disabled.
**Example:** ```nginx app_protect_dos_security_log_enable on; ``` ### Security log directive (`app_protect_dos_security_log`) This directive has two string arguments. First argument is the configuration file path, i.e. `/etc/app_protect_dos/log-default.json`. Second argument is the destination (the location which the events will be sent to). The destination can be one of three options: - `syslog:server={ip}:{port}`, i.e. `syslog:server=1.2.3.4:3000` - `stderr` (**default**) - `{absolute_file_path}`, i.e. `/shared/dos_sec_logger.log` Implemented according to: [F5 DoS for NGINX Security Log](/nap-dos/monitoring/security-log.md) **Note:** - When using stderr, make sure that the process `admd` is not redirecting the stderr output to file. - When using the Docker `entrypoint.sh` startup script from the admin guide, make sure that it doesn’t redirect stderr. **Examples:** - **Syslog:** ```nginx app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" syslog:server=1.2.3.4:5000; ``` - **File:** ```nginx app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" /shared/logger.log; ``` - **Stderr:** ```nginx app_protect_dos_security_log "/etc/app_protect_dos/log-default.json" stderr; ``` While `/etc/app_protect_dos/log-default.json` is: ```json { "filter": { "traffic-mitigation-stats": "all", "bad-actors": "top 10", "attack-signatures": "top 10" } } ``` ### Liveness probe directive (`app_protect_dos_liveness`) This directive has 3 arguments. | First argument | Second argument | Third argument | | :-------------- | :--------------- | :-------------- | | [on\|off] depending if this feature should be enabled or disabled. | URI Syntax is: `uri:___` | Port Syntax is: `port:____` | **Note:** Second and Third arguments are optional; if one or more is not written, the default will take place. If liveness is enabled, a request with URI and PORT that matches the probe configuration (i.e. `/app_protect_dos_liveness:8090`) will be answered with RC 200 "Alive" by our NGINX module, without being counted or pass to other handlers nor the backend server. Any other response will indicate that our NGINX module (F5 DoS for NGINX) has not received the request (possibly means that NGINX is down). **Example:** ```nginx app_protect_dos_liveness on uri:/liveness port:8090; ``` ### Readiness probe directive (`app_protect_dos_readiness`) This directive has 3 arguments. | First argument | Second argument | Third argument | | :-------------- | :--------------- | :-------------- | | [on\|off] depending if this feature should be enabled or disabled. | URI Syntax is: `uri:___` | Port Syntax is: `port:____` | **Note:** Second and Third arguments are optional; if one or more is not written, the default will take place. If readiness is enabled, a request with URI and PORT that matches the probe configuration (i.e. `/app_protect_dos_readiness:8090`) will be answered with RC 200 "Ready" or RC 503 "Not Ready" by our NGINX module, without being counted or pass to other handlers nor the backend server. Any other response will indicate that our NGINX module (F5 DoS for NGINX) has not received the request (possibly means that NGINX is down). RC 200 "Ready" will occur if two conditions are met: 1. NGINX worker successfully connected to the global shared memory block 2. ADMD process is running (and not stuck) **Example:** ```nginx app_protect_dos_readiness on uri:/readiness port:8090; ``` ### Arbitrator FQDN directive (`app_protect_dos_arb_fqdn`) Arbitrator FQDN directive has one argument which is the `FQDN/IP`. The argument is the FQDN to the desired Arbitrator. **Examples:** FQDN: ```nginx app_protect_dos_arb_fqdn svc-appprotect-dos-arb.arb.svc.cluster.local; ``` IP address: ```nginx app_protect_dos_arb_fqdn 192.168.1.10; ``` ### API directive (`app_protect_dos_api`) This directive is used to enable the App Protect DoS monitoring capability via REST API.
The REST API interface provides extended metrics information of the Protected Objects. It can be used by sending REST API requests manually or by using the App Protect DoS dashboard page. For more information refer to [F5 DoS for NGINX Live Activity Monitoring](/nap-dos/monitoring/live-activity-monitoring.md) **Example:** ```nginx http { # ... server { listen 192.168.1.23; # ... location /api { app_protect_dos_api; allow 192.168.1.0/24; deny all; } location = /dashboard-dos.html { root /usr/share/nginx/html; } } } ``` ### Accelerated mitigation directive (`app_protect_dos_accelerated_mitigation`) This directive is used to enable or disable App Protect DoS L4 accelerated mitigation.
syn_drop is an optional parameter; the default value is "off".
syn_drop=on mode is applicable for plane HTTP services or HTTPS when the `tls_fingerprint` feature is disabled. Refer to policy parameter "tls_fingerprint" in [Policy directive](#policy-directive-app_protect_dos_policy_file). In syn_drop mode, the SYN packet of detected bad actors will be dropped. syn_drop mode is recommended for the deployments of F5 DoS for NGINX at the perimeter network or behind L3 load balancer. Using this mode when F5 DoS for NGINX is deployed behind L4/L7 load balancer may result in the load balancer’s starvation during an attack. **Note:** To use this directive you need to install the eBPF package. For more information about eBPF, you can read the [Accelerating DDoS Mitigation with eBPF in F5 DoS for NGINX](https://www.f5.com/company/blog/nginx/accelerating-ddos-mitigation-with-ebpf-in-f5-nginx-app-protect-dos) article. **Example:** ```nginx app_protect_dos_accelerated_mitigation on syn_drop=on; ``` ### Access File directive (`app_protect_dos_access_file`) The `app_protect_dos_access_file` directive defines an allowlist policy from a specified file.
This enables specifying IP addresses or ranges that should never be blocked.
The format of the file is the same as used in F5 WAF for NGINX, making it easy to reuse existing WAF policies with defined allowlist IPs.

The directive is optional. If not written, then the allowlist feature is disabled.

The file should include a list of IP addresses or ranges in JSON format. Both IPv4 and IPv6 addresses are supported.
IPv4 addresses are in the format "a.b.c.d" where each component is a decimal number in the range 0-255.
IPv6 addresses are in the format "h1:h2:h3:h4:h5:h6:h7:h8" where each component is a hex number in the range 0x0-0xffff. Any contiguous range of zero elements can be omitted and replaced by "::".
IPv4 and IPv6 masks are written in the format "IP/xxx" (for example: /24), indicating the number of significant bits.

The JSON file should include the ipAddress field for specifying IP addresses or ranges, and the blockRequests field set to "transparent". The file can also include $ref to reference additional files containing more IP addresses.

Additionally, a second format is supported where the mask is specified in a dedicated field `ipMask`. The mask should be written in the standard subnet notation for IPv4 and IPv6 addresses. In this format, the `blockRequests` field should have a value of "never" instead of "transparent".

**Example:** ```nginx app_protect_dos_access_file "/etc/app_protect_dos/allowlist.json"; ``` **Example content of /etc/app_protect_dos/allowlist.json:** ```nginx { "policy": { "ip-address-lists": [ { "ipAddresses": [ { "ipAddress": "1.1.1.1" }, { "ipAddress": "1.1.1.1/32" }, { "ipAddress": "3.3.3.0/24" }, { "ipAddress": "2023::4ef3/128" }, { "ipAddress": "2034::2300/120" } ], "blockRequests": "transparent" }, { "$ref": "/etc/app_protect_dos/additional_ips.json", "blockRequests": "transparent" } ] } } ``` **Example content of /etc/app_protect_dos/additional_ips.json:** ```nginx { "ipAddresses": [ { "ipAddress": "2.2.2.2/32" }, { "ipAddress": "4.4.4.0/24" } ] } ``` **Example content with second format:** ```nginx { "policy":{ "whitelist-ips":[ { "ipAddress":"2034::2300", "ipMask":"ffff:ffff:ffff:ffff:ffff:ffff:ffff:ff00", "blockRequests":"never" }, { "blockRequests":"never", "ipAddress":"4.4.4.0", "ipMask":"255.255.255.0" } ] } } ``` -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 4.0" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-4.0/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v4.0. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 4.0 January 31, 2023 ### New Features - Distributed Denial of Service (DDoS) protection feature for WebSocket services.
Refer to the [Configuration Example](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#monitor-directive-app_protect_dos_monitor) for WebSocket services here.
- DDoS protection against slow attacks has been improved using machine learning algorithm on all types of traffic. - `app_protect_dos_monitor` directive, which monitors the proxied server, supports a new type of protocol - "WebSocket". ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-28+4.0.1-1.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-28+4.0.1-1.el8.ngx.x86_64.rpm ##### Debian 11 - app-protect-dos_28+4.0.1.-1~bullseye_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_28+4.0.1-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_28+4.0.1-1~focal_amd64.deb ##### Alpine 3.15 - app-protect-dos-28.4.0.1-r1.apk #### NGINX Plus - NGINX Plus R28 ### Important Notes - WebSocket protection requires active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - Installing accelerated mitigation feature (install `app-protect-dos-ebpf`) configures `nginx` and `admd` to run with root privileges. - Support for `proxy_protocol` configuration: `proxy_protocol` monitor parameter should be used when the `listen` directive of the correspondent server block contains the `proxy_protocol` parameter. - If F5 WAF for NGINX is installed, app protect should be disabled for the location of DoS Live Activity Monitoring API. For example: ```shell location /api { app_protect_enable off; app_protect_dos_api; } ``` - Port configuration in `app_protect_dos_monitor` should correspond to the port, the server listens to. Misconfiguration can potentially cause a false attack declaration. For example: ```shell server { listen 8080; location / { app_protect_dos_monitor "myservice.com:8080"; } } ``` - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for the attack to be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Monitor directive `app_protect_dos_monitor` with `proxy_protocol` parameter can not be configured on Ubuntu 18.04. As a result, gRPC and HTTP/2 DoS protection for `proxy_protocol` configuration is not supported. - Slow attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "Logs Overview" url: https://docs.nginx.com/nginx-app-protect-dos/monitoring/types-of-logs/index.md description: Learn about the F5 DoS for NGINX Logs Overview. -------------------------------------------------------------------------------- There are 4 types of logs corresponding to App Protect DoS: - [Security Log](#security-log): The general picture of the site and how App Protect DoS processed it, including anomalies and signatures found. - [Operation Log](#operation-log): Events such as configuration errors or warnings. - [Debug Logs](#debug-log): Technical messages at different levels of severity used to debug and resolve incidents and error behaviors. - [Request Logging](#request-log): F5 DoS for NGINX adds information to each request logged to NGINX's access logging mechanism. **Note:** NGINX does not have audit logs in the sense of *"**who** did **what**"*. This can be done either from the orchestration system controlling NGINX (such as NGINX Controller) or by tracking the configuration files and the systemd invocations using Linux tools. |Type|Log Configuration| Configuration Contexts| File Destination| Syslog Destination | |----|-----------------|-----------------------|-----------------|--------------------| | Debug | Log file name is the redirection in the invocation of the `admd` command line in the start script | Global (not part of `nginx.conf`)|Yes. Log file is in /var/log/adm/admd.log directory. There is currently no file rotation capability available for this log.| No | | Operation | `error_log` directive, part of core NGINX | `nginx.conf` - global | Yes, NGINX error log | Yes, NGINX error log | |Request |NGINX has two directives for the access log:
- **access_log** - to turn [on\|off]
- **log_format** - to specify the required information regarding each request

F5 DoS for NGINX has several variables that can be added to the log_format directive, such as $app_protect_dos_outcome.

For more information refer to [F5 DoS for NGINX Access Log](/nap-dos/monitoring/access-log.md) | `nginx.conf` - global| Yes, NGINX access log | Yes, NGINX access log | | Security | F5 DoS for NGINX has two directives in `nginx.conf`:
- app_protect_dos_security_log_enable to turn logging [on\|off]
- app_protect_dos_security_log to set it's logging configuration and destination

For more information refer:
- **Configuration**: [App Protect DoS - Directives and Policy](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md)
- **Usage**: [F5 DoS for NGINX - Security Log](/nap-dos/monitoring/security-log.md) | `nginx.conf`: http, server, location | Yes, either stderr, or an absolute path to a local file are supported | Yes | ## Security Log The security logs contain information about the status of the protected objects. It gives a general picture about each protected object in terms of traffic intensity, health of the backend server, learning and mitigations. For more information refer to [F5 DoS for NGINX Security Log](/nap-dos/monitoring/security-log.md) documentation. ## Operation Log The operation logs consists of system operational and health events. The events are sent to the NGINX error log and are distinguished by the `APP_PROTECT_DOS` prefix followed by JSON body. The log level depends on the event: success is usually indicated by `notice`, while failure is indicated by `error`. The timestamp is inherent in the error log. For more information refer to [App Protect DoS Operation Log](/nap-dos/monitoring/operation-log.md) documentation. ## Request Log Access log is NGINX’s request log mechanism. It is controlled by two directives. ### log_format This directive determines the format of the log messages using predefined variables. App Protect DoS will enrich this set of variables with several security log attributes that are available to be included in the `log_format`. If `log_format` is not specified then the built-in format `combined` is used but, because that format does not include the extended App Protect DoS variables, this directive must be used when the user wants to add App Protect DoS information to the log. ### access_log This directive determines the destination of the `access_log` and the name of the format. The default is the file `/var/log/nginx/access.log` using the combined format. In order to use the custom format that includes the F5 DoS for NGINX variables, use this directive with the name of the desired format. ### App Protect DoS Variables These are the variables added to Access Log. They are a subset of the Security log attributes. The Security log names are prefixed with `$app_protect_dos`.
For more information refer to [F5 DoS for NGINX Access Log](/nap-dos/monitoring/access-log.md) ## Debug Log - F5 DoS for NGINX The F5 DoS for NGINX Debug log is used to troubleshoot the functionality of the product.
The path of the log is at a fixed location: `/var/log/adm/admd.log`. There are several log levels - `error`, `warning`, `info` and `debug`. The default is `info`. In order to change the log level at run time, the following command can be called: ```shell admd -l DEBUG_LEVEL ``` **Note:** `nginx.conf` does not refer to the F5 DoS for NGINX debug log configuration neither directly nor indirectly. ## NGINX Error log The NGINX Error log is used to troubleshoot the configuration portion of F5 DoS for NGINX. The file is called `error.log` and its path and debug level is determined in `nginx.conf` by the directive `error_log`.
For example: ```shell error_log /var/log/nginx/error.log debug; ``` -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 3.1" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-3.1/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v3.1. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 3.1 November 29, 2022 In this release, F5 DoS for NGINX supports NGINX Plus R28. ### New Features - Support for NGINX Plus R28. - F5 DoS for NGINX can be deployed behind L4/L7 load balancers when L4 protection is enabled. - Fixed the issue: Installing F5 DoS for NGINX with L4 (eBPF) mitigation behind L4/L7 load balancer may result in the load balancer's starvation during an attack. - Support for [proxy_protocol](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#monitor-directive-app_protect_dos_monitor) configuration for server health monitoring.
Previously, server health monitoring could not be used when the listen directive of the correspondent server block contained the `proxy_protocol` parameter.
This disallowed using F5 DoS for NGINX for Denial of Service (DoS) protection for HTTP2 and gRPC protected objects in the `proxy_protocol` configuration. ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-28+3.1.7-1.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-28+3.1.7-1.el8.ngx.x86_64.rpm ##### Debian 11 - app-protect-dos_28+3.1.7-1~bullseye_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_28+3.1.7-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_28+3.1.7-1~focal_amd64.deb ##### Alpine 3.15 - app-protect-dos-28.3.1.7-r1.apk #### NGINX Plus - NGINX Plus R28 ### Important Notes - Installing L4 accelerated mitigation feature (install `app-protect-dos-ebpf`) configures `nginx` and `admd` to run with root privileges. - Support for `proxy_protocol` configuration: `proxy_protocol` monitor parameter should be used when the listen directive of the correspondent server block contains the `proxy_protocol` parameter. - If F5 WAF for NGINX is installed, app protect should be disabled for the location of DoS Live Activity Monitoring API. For example: ```shell location /api { app_protect_enable off; app_protect_dos_api; } ``` - Misconfiguration of `app_protect_dos_monitor` potentially can cause a false attack declaration. Port configuration should correspond to the port the server listens to. For example: ```shell server { listen 8080; location / { app_protect_dos_monitor "myservice.com:8080"; } } ``` - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Monitor directive (app_protect_monitor) with the `proxy_protocol` parameter can not be configured on Ubuntu 18.04. As a result, gRPC and HTTP/2 DoS protection for `proxy_protocol` configuration is not supported. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "Live Activity Monitoring" url: https://docs.nginx.com/nginx-app-protect-dos/monitoring/live-activity-monitoring/index.md description: Learn about the F5 DoS for NGINX Live Activity Monitoring. -------------------------------------------------------------------------------- ![F5 DoS for NGINX Dashboard](/dashboard/dos-tab.png) ## Overview F5 DoS for NGINX offers a variety of application monitoring tools: - **App Protect DoS Dashboard Page**: This dynamic interface provides real-time monitoring and details of Protected Objects. - **F5 DoS for NGINX REST API**: This interface offers comprehensive metrics about the Protected Objects. ## Prerequisites - NGINX Plus R26 or later is required for accessing the F5 DoS for NGINX REST API and the DoS Dashboard. ## API Configuration Steps To activate the API: - Define a `server` block within the `http` context to handle the API: ```nginx http { server { # insert your API configuration here } } ``` - Create a `location` for API requests and include the `app_protect_dos_api` directive: ```nginx http { # ... server { listen 192.168.1.23; # ... location /api { app_protect_dos_api; # ... } } } ``` - Limit API location access, for example, permit only local network access using `allow` and `deny` directives: ```nginx http { # ... server { listen 192.168.1.23; # ... location /api { app_protect_dos_api; allow 192.168.1.0/24; deny all; } } } ``` - Implement HTTP basic authentication to restrict access to PATCH, POST, and DELETE methods to certain users: ```nginx http { # ... server { listen 192.168.1.23; # ... location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } app_protect_dos_api; allow 192.168.1.0/24; deny all; } } } ``` - Enable the DoS Dashboard by defining the /dashboard-dos.html location, which is typically in the root directory: ```nginx http { # ... server { listen 192.168.1.23; # ... location /api { limit_except GET { auth_basic "NGINX Plus API"; auth_basic_user_file /path/to/passwd/file; } app_protect_dos_api; allow 192.168.1.0/24; deny all; } location = /dashboard-dos.html { root /usr/share/nginx/html; } } } ``` ## Use the Dashboard ### Access the Dashboard To view the Dashboard, enter the corresponding address in your browser's address bar.
For example, "http://192.168.1.23/dashboard-dos.html" will show the Dashboard page located in /usr/share/nginx/html as indicated in the root directive. ### DoS Tab Overview The **DoS** tab displays real-time statistics, settings, and traffic graphs for each Protected Object.
In multi-instance environments with an arbitrator, these statistics will be combined. | Metric name | Values | Description | |:--------------- |:------- |:-----------| | Name | - | The name of the Protected Object, as defined by the `app_protect_dos_name` directive (or auto-generated if not present) | | Health | [good\|bad] | The health of the backend server, as defined by the `uri` argument of the `app_protect_dos_monitor` directive | | Under Attack | [yes\|no] | Whether the Protected Object is under attack or not | | Req/s | - | Number of incoming requests per second | | Mitigations/s | - | Number of mitigated requests per second | | Requests | - | Total number of incoming requests | | Mitigations | - | Total number of mitigated requests | | Learning | [ready\|ba only\|not ready] | Whether F5 DoS for NGINX collected enough data to protect the Protected Object. Note: `ba only` means that request thresholds were learned, but full request sampling is not complete. In this status, F5 DoS for NGINX will only detect bad actors (by IP or IP+TLS fingerprint) and will not utilize mitigation signatures. | | Protocol | [http1\|http2\|grpc] | As defined by the `protocol` argument of the `app_protect_dos_monitor` directive | | Mitigation Mode | [standard\|conservative\|none] | As defined by the `mitigation_mode` object in the JSON policy file from the `app_protect_dos_policy_file` directive | | Signatures | [on\|off] | As defined by the `signatures` object in the JSON policy file from the `app_protect_dos_policy_file` directive. Values - on/off | | Bad Actors | [on\|off] | As defined by the `bad_actors` object in the JSON policy file from the `app_protect_dos_policy_file` directive | | Automation Tools Detection | [on\|off] | As defined by the `automation_tools_detection` object in the JSON policy file from the `app_protect_dos_policy_file` directive | | TLS Fingerprint| [on\|off] | As defined by the `tls_fingerprint` object in the JSON policy file from the `app_protect_dos_policy_file` directive |
### Graph Representation The displayed graph is a stacked composition, encompassing two specific metrics: `Passthrough Requests` and `Mitigations`, with both metrics measured on a per-second basis. - `Passthrough Requests` represents the count of requests successfully forwarded to the backend server. - `Mitigations` indicates the quantity of requests that have been mitigated.
The sum of these two metrics at any given moment provides the aggregate count of incoming requests. ### Status Color Indicators - **Green**: Indicates a normal state, with no attack and optimal server health. - **Yellow**: Indicates the presence of either an active attack or diminished server health. - **Red**: Indicates a severe scenario, where the object is experiencing an attack combined with poor health. ### Configure Dashboard Options You can configure the dashboard by clicking the Gear button in the Tabs menu.
**Update every N sec** - updates the Dashboard data after the specified number of seconds, default is 1 second.

## Use the REST API Statistics of your server infrastructure can be managed with the REST API interface. The API is based on standard HTTP requests: statistics can be obtained with `GET` requests. The requests are sent in the JSON format that allows you to connect the stats to monitoring tools or dashboards that support JSON. The status information of any element can be accessed with a slash-separated URL. The URL may look as follows: `http://192.168.1.23/api/dos/1/protected_objects` where: - `/api` represents the configured endpoint in your NGINX setup for API access. - `/dos` differentiates the App Protect DoS API from other NGINX Plus APIs. - `/1` indicates the API version you are using, with the current version being 1. - `/protected_objects` points to the specific resource or data you want to access. Responses from the API are in JSON format for easy parsing and analysis. To view all primary endpoints, use a GET request via 'curl' in the terminal: ```shell curl -s 'http://192.168.1.23/api/dos/1/' | json_pp ``` This command returns a JSON array listing the main access points, like: ```json ["protected_objects"] ``` For obtaining detailed statistics about a particular endpoint, execute a similar GET request: ```shell curl -s 'http://192.168.1.23/api/dos/1/protected_objects' | json_pp ``` ### APIs overview #### Endpoints | Path | Description | |:------ |:-----------| | / | Return list of root endpoints | | /protected_objects/ | Return statistics of all Protected Objects | | /protected_objects/{protectedObjectName} | Return statistics of a Protected Object | An example response from the `/protected_objects` endpoint could be: ```json { "po_1": { "attack": false, "health": 0.50, "rps": 12, "mps": 0, "requests": 500123, "passthrough": 260023, "mitigations": 240100, "learning": "ready", "protocol": "http1", "mode": "standard", "sig": true, "ba": true, "auto_tools": true, "tls_fp": true }, "po_http2": { "attack": false, "health": 0.50, "rps": 20, "mps": 0, "requests": 500123, "passthrough": 260023, "mitigations": 240100, "learning": "not_ready", "protocol": "http2", "mode": "conservative", "sig": true, "ba": true, "auto_tools": true, "tls_fp": true }, "po_grpc": { "attack": false, "health": 0.50, "rps": 25, "mps": 0, "requests": 6000123, "passthrough": 2599123, "mitigations": 3401000, "learning": "ready", "protocol": "grpc", "mode": "standard", "sig": true, "ba": true, "auto_tools": true, "tls_fp": true } } ``` -------------------------------------------------------------------------------- title: "Security Log" url: https://docs.nginx.com/nginx-app-protect-dos/monitoring/security-log/index.md description: Learn about the F5 DoS for NGINX Security Log. -------------------------------------------------------------------------------- ## Overview Security logs contain information about the status of the protected objects. It gives a general picture about each protected object in terms of traffic intensity, health of the backend server, learning and mitigations. There are several types of logs, each contains different information and published either periodically or upon an important event. ### Dictionary The following table lists all the possible fields in the logs and their meaning. | Field | Type | Meaning | |----------------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `date_time` | string | the date and time of the event | | `product` | string | always set to `app-protect-dos` | | `product_version` | string | F5 DoS for NGINX version | | `unit_hostname` | string | host name of the app-protect-dos instance | | `instance_id` | string | instance ID: container id from `/proc/self/cgroup`or hostname if container is is not available | | `vs_name` | string | A unique identifier (representing the protected object's name) of the location in the `nginx.conf` file that this request is associated with. It contains the line number of the containing server block in `nginx.conf`, the server name, a numeric discriminator that distinguishes between multiple entries within the same server, and the location name.
For example: `34-mydomain.com:0-~/.*php(2)`. | | `dos_attack_id`| integer | unique attack IP per unit_hostname | | `attack_event` | string | Event name as it appears in remote logger. | | `stress_level` | float | a number from 0 to ... that reflects stress level. | | `learning_confidence` | string | the possible values are **not ready/bad actors only/ready** | | `baseline_dps` | integer | learned datagrams per second (DPS) | | `incoming_dps` | integer | current datagrams per second (DPS) | | `incoming_rps` | integer | current RPS (requests per second) | | `successful_tps` | integer | successful TPS (successful requests per second - Any RC but 5xx) | | `allowlist_rps` | integer | allowlist requests per second | | `unsuccessful_rps` | integer | unsuccessful requests per second (passed to server and not responded: `reset / timeout / 5xx` | | `incoming_datagrams` | integer | incremental number of incoming datagrams | | `incoming_requests` | integer | incremental number of incoming requests | | `allowlist_requests` | integer | incremental number of allowlist requests | | `successful_responses` | integer | incremental number of successful responses | | `unsuccessful_requests` | integer | incremental number of unsuccessful requests (passed to server and not responded: `reset / timeout / 5xx` | | `active_connections`| integer | current number of active server connections | | `threshold_dps` | float | global rate DPS threshold | | `threshold_conns` | float | active connections threshold | | `mitigated_bad_actors`
`redirect_bad_actor`
`challenge_bad_actor`
`block_bad_actor` | integer | incremental number of mitigated bad actors. Increments upon any type of bad actors mitigations.
incremental number of http redirections sent to detected bad actors
incremental number of JS challenges sent to detected bad actors
incremental number of blocked bad actors | | `mitigated_by_signatures`
`redirect_signature`
`challenge_signature`
`block_signature` | integer | incremental number of requests mitigated by signatures. Increments upon any type of signatures mitigations.
incremental number of http redirections sent to clients when requests match a signature.
incremental number of JS challenges sent to clients when requests match a signature.
incremental number of blocked requests when requests match a signature. | | `mitigated_by_global_rate`
`redirect_global`
`challenge_global`
`block_global` | integer | incremental number of requests mitigated by global_rate. Increments upon any type of global rate mitigations.
incremental number of http redirections sent to clients upon global rate mitigation.
incremental number of JS challenges sent to clients upon global rate mitigation.
incremental number of blocked requests upon global rate mitigation. | | `mitigated_slow`
`redirect_slow`
`challenge_slow`
`block_slow`| integer | incremental number of mitigated slow requests. Increments upon any type of slow requests mitigations.
incremental number of http redirections sent to clients upon slow request mitigation.
incremental number of JS challenges sent to clients upon slow request mitigation.
incremental number of blocked slow requests. | | `mitigated_connections` | integer | incremental number of mitigated by connections mitigation | | `mitigated_bad_actors_l4` | integer | incremental number of mitigated by L4 accelerated mitigation | | `mitigated_bad_actors_rps`
`redirect_bad_actor_rps`
`challenge_bad_actor_rps`
`block_bad_actor_rps` | integer | mitigated_bad_actors rps. Includes any type of bad actors mitigations.
http redirections per second sent to detected bad actors.
JS challenges per second sent to detected bad actors.
blocked bad actors per second. | | `mitigated_by_signatures_rps`
`redirect_signature_rps`
`challenge_signature_rps`
`block_signature_rps` | integer | mitigated_signatures rps. Includes any type of signatures mitigations.
http redirections sent per second to clients when requests match a signature.
JS challenges per second sent to clients when requests match a signature.
blocked requests per second when requests match a signature. | | `mitigated_slow_rps`
`redirect_slow_rps`
`challenge_slow_rps`
`block_slow_rps` | integer | mitigated slow requests per second. Includes any type of slow requests mitigations.
http redirections per second sent to clients upon slow request mitigation.
JS challenges per second sent to clients upon slow request mitigation.
blocked slow requests per second. | | `mitigated_by_global_rate_rps`
`redirect_global_rps`
`challenge_global_rps`
`block_global_rps` | integer | mitigated_global_rate rps. Includes any type of global rate mitigations.
http redirections per second sent to clients upon global rate mitigation.
JS challenges per second sent to clients upon global rate mitigation.
blocked requests per second upon global rate mitigation. | | `mitigated_bad_actors_l4_rps` | integer | blocked requests per second when mitigated by L4 accelerated mitigation | | `mitigated_connections_rps` | integer | mitigated_connections rps | | `source_ip`
`tls_fp`
`impact_rps` | string
string
integer | ip address of the detected bad actor `1.1.1.1`
TLS Fingerprint of the bad actor
RPS created by bad actor in the time of the detection **(to be calculated as a max hitcount in AMT / 10)** | | `new_bad_actors_detected`
`bad_actors` | integer | the number of newly detected bad actors
the number of bad actors | | `signature`
`signature_id`
`signature_efficiency`
`signature_accuracy`| string
integer
float
float | signature string `http.request.method eq GET and http.uri_parameters eq 6`
unique signature ID per unit_host
estimated efficiency upon signature detection: percentage of bad traffic covered by the signature
estimated accuracy upon signature detection: percentage of learned good traffic NOT covered by the signature | ## Events ### 1a. Attack notification Reports about the start and end of an attack, as well as major parameters of ongoing attacks. a. Example: **Attack Started** ```shell date_time="Oct 05 2021 08:01:00", product="app-protect-dos", product_version="25+1.78.0-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="129c76", vs_name="example.com/", dos_attack_id="1", attack_event="Attack started", stress_level="1.00", learning_confidence="Ready", baseline_dps="17", incoming_dps="181", incoming_rps="181", successful_tps="0", allowlist_rps="0", unsuccessful_rps="0", incoming_datagrams="8576", incoming_requests="8576", allowlist_requests="162", successful_responses="5265", unsuccessful_requests="0", active_connections="58", threshold_dps="41.60", threshold_conns="41.60", mitigated_bad_actors="0", mitigated_by_signatures="0", mitigated_by_global_rate="0", mitigated_bad_actors_l4="0", mitigated_slow="0", redirect_global="0", redirect_bad_actor="0", redirect_signature="0", redirect_slow="0", challenge_global="0", challenge_bad_actor="0", challenge_signature="0", challenge_slow="0", block_global="0", block_bad_actor="0", block_signature="0", block_slow="0", mitigated_connections="0", mitigated_bad_actors_rps="0", mitigated_by_signatures_rps="0", mitigated_by_global_rate_rps="0", mitigated_bad_actors_l4_rps="0", mitigated_slow_rps="0", redirect_global_rps="0", redirect_bad_actor_rps="0", redirect_signature_rps="0", redirect_slow_rps="0", challenge_global_rps="0", challenge_bad_actor_rps="0", challenge_signature_rps="0", challenge_slow_rps="0", block_global_rps="0", block_bad_actor_rps="0", block_signature_rps="0", block_slow_rps="0", mitigated_connections_rps="0", ``` b. Example: **Attack Ended** ```shell date_time="Oct 05 2021 08:06:21", product="app-protect-dos", product_version="25+1.78.0-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="129c76", vs_name="example.com/", dos_attack_id="1", attack_event="Attack ended", stress_level="0.50", learning_confidence="Ready", baseline_dps="12", incoming_dps="0", incoming_rps="0", successful_tps="0", allowlist_rps="0", unsuccessful_rps="0", incoming_datagrams="226566", incoming_requests="226566", allowlist_requests="1632", successful_responses="7760", unsuccessful_requests="0", active_connections="0", threshold_dps="2121.60", threshold_conns="2121.60", mitigated_bad_actors="94488", mitigated_by_signatures="117361", mitigated_by_global_rate="2861", mitigated_bad_actors_l4="62788", mitigated_slow="0", redirect_global="2861", redirect_bad_actor="94488", redirect_signature="117361", redirect_slow="0", challenge_global="0", challenge_bad_actor="0", challenge_signature="0", challenge_slow="0", block_global="0", block_bad_actor="0", block_signature="0", block_slow="0", mitigated_connections="0", mitigated_bad_actors_rps="0", mitigated_by_signatures_rps="0", mitigated_by_global_rate_rps="0", mitigated_bad_actors_l4_rps="0", mitigated_slow_rps="0", redirect_global_rps="0", redirect_bad_actor_rps="0", redirect_signature_rps="0", redirect_slow_rps="0", challenge_global_rps="0", challenge_bad_actor_rps="0", challenge_signature_rps="0", challenge_slow_rps="0", block_global_rps="0", block_bad_actor_rps="0", block_signature_rps="0", block_slow_rps="0", mitigated_connections_rps="0", ``` ### 1b. Traffic/Mitigation summary stats Reported periodically, providing aggregated statistics per protected object.
This corresponds to the metrics reported on the main Grafana screen. a. Example: **No Attack** ```shell date_time="Oct 05 2021 07:54:29", product="app-protect-dos", product_version="25+1.78.0-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="129c76", vs_name="example.com/", dos_attack_id="0", attack_event="No Attack", stress_level="0.50", learning_confidence="Not ready", baseline_dps="19", incoming_dps="9", incoming_rps="9", successful_tps="10", allowlist_rps="1", unsuccessful_rps="0", incoming_datagrams="678", incoming_requests="678", allowlist_requests="52", successful_responses="678", unsuccessful_requests="0", active_connections="0", threshold_dps="2121.60", threshold_conns="2121.60", mitigated_bad_actors="0", mitigated_by_signatures="0", mitigated_by_global_rate="0", mitigated_bad_actors_l4="0", mitigated_slow="0", redirect_global="0", redirect_bad_actor="0", redirect_signature="0", redirect_slow="0", challenge_global="0", challenge_bad_actor="0", challenge_signature="0", challenge_slow="0", block_global="0", block_bad_actor="0", block_signature="0", block_slow="0", mitigated_connections="0", mitigated_bad_actors_rps="0", mitigated_by_signatures_rps="0", mitigated_by_global_rate_rps="0", mitigated_bad_actors_l4_rps="0", mitigated_slow_rps="0", redirect_global_rps="0", redirect_bad_actor_rps="0", redirect_signature_rps="0", redirect_slow_rps="0", challenge_global_rps="0", challenge_bad_actor_rps="0", challenge_signature_rps="0", challenge_slow_rps="0", block_global_rps="0", block_bad_actor_rps="0", block_signature_rps="0", block_slow_rps="0", mitigated_connections_rps="0", ``` b. Example: **Under Attack** ```shell date_time="Oct 05 2021 08:02:35", product="app-protect-dos", product_version="25+1.78.0-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="129c76", vs_name="example.com/", dos_attack_id="1", attack_event="Under Attack", stress_level="0.50", learning_confidence="Ready", baseline_dps="12", incoming_dps="893", incoming_rps="893", successful_tps="12", allowlist_rps="1", unsuccessful_rps="0", incoming_datagrams="87823", incoming_requests="87823", allowlist_requests="1523", successful_responses="5736", unsuccessful_requests="0", active_connections="1", threshold_dps="92.40", threshold_conns="92.40", mitigated_bad_actors="0", mitigated_by_signatures="75137", mitigated_by_global_rate="2861", mitigated_bad_actors_l4="62788", mitigated_slow="0", redirect_global="2861", redirect_bad_actor="0", redirect_signature="75137", redirect_slow="0", challenge_global="0", challenge_bad_actor="0", challenge_signature="0", challenge_slow="0", block_global="0", block_bad_actor="0", block_signature="0", block_slow="0", mitigated_connections="0", mitigated_bad_actors_rps="0", mitigated_by_signatures_rps="879", mitigated_by_global_rate_rps="0", mitigated_bad_actors_l4_rps="0", mitigated_slow_rps="0", redirect_global_rps="0", redirect_bad_actor_rps="0", redirect_signature_rps="879", redirect_slow_rps="0", challenge_global_rps="0", challenge_bad_actor_rps="0", challenge_signature_rps="0", challenge_slow_rps="0", block_global_rps="0", block_bad_actor_rps="0", block_signature_rps="0", block_slow_rps="0", mitigated_connections_rps="0", ``` ### 2. Bad actor detection/expiration Reports F5 DoS for NGINX decisions regarding bad actors. a. Example: **Bad Actor Detection** ```shell date_time="Apr 29 2021 14:03:01", product="app-protect-dos", product_version="23+1.54.1-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="d9a6d8", vs_name="example.com/", dos_attack_id="1", attack_event="Bad actor detection", source_ip="5.5.5.9", impact_rps="30", ``` b. Example: **Bad Actor Expired** ```shell date_time="Apr 29 2021 14:05:29", product="app-protect-dos", product_version="23+1.54.1-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="d9a6d8", vs_name="example.com/", dos_attack_id="0", attack_event="Bad actor expired", source_ip="5.5.5.10", impact_rps="12", ``` ### 3. Attack signatures Reports F5 DoS for NGINX decisions regarding signatures.
Example: **Attack Signature Detected** ```shell date_time="Apr 29 2021 14:02:56", product="app-protect-dos", product_version="23+1.54.1-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="d9a6d8", vs_name="example.com/", dos_attack_id="1", attack_event="Attack signature detected", signature="(http.user_agent_header_exists eq true) and (http.accept contains other-than(application|audio|message|text|image|multipart)) and (http.unknown_header_exists eq true) and (http.headers_count neq 10) and (http.x_forwarded_for_header_exists eq false) and (http.uri_parameters eq 1) and (http.uri_len between 48-63) and (http.accept_header_exists eq true) and (http.hdrorder not-hashes-to 55) and (http.connection_header_exists eq true) and (http.accept_encoding_header_exists eq true) and (http.request.method eq reserved) and (http.cookie_header_exists eq true) and (http.uri_file hashes-to 7) and (http.host_header_exists eq true)", signature_id="809655398", signature_efficiency="72.00", signature_accuracy="100.00", ``` ### 4. Bad actors detection information Provides detailed information about bad actors.
Example: **Bad Actors Detected** ```shell date_time="Apr 29 2021 14:02:00", product="app-protect-dos", product_version="23+1.54.1-1.el7.ngx", unit_hostname="localhost.localdomain", instance_id="d9a6d8", vs_name="example.com/", dos_attack_id="1", attack_event="Bad actors detected", new_bad_actors_detected="2", bad_actors="2", ``` ## Security Log Configuration File The file is in JSON format.
### Filter
| Element | Description | Type/Values| Default | |----------|--------------| ---------- | -----------| |traffic-mitigation-stats| This filter element refers to [Traffic/Mitigation summary stats](#1b-trafficmitigation-summary-stats).| **Enumerated values:**
- **all**
- **none**| `all` | |bad-actors| This filter element refers to [Bad actor detection/expiration](#2-bad-actor-detectionexpiration), every 10 seconds.| **Enumerated values:**
- **all**
- **none**
- **top N**| `top 10` | |attack-signatures| This filter element refers to [Attack Signatures](#3-attack-signatures), every 10 seconds.| **Enumerated values:**
- **all**
- **none**
- **top N**| `top 10` | Example: ```json { "filter": { "traffic-mitigation-stats": "all", "bad-actors": "top 100", "attack-signatures": "top 100" } } ``` -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 3.0" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-3.0/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v3.0. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 3.0 September 21, 2022 ### New Features - L4 mitigation (with eBPF) - DoS Live Activity Monitoring with requests mitigation graphs - DoS Live Activity Monitoring support for multi-instances F5 DoS for NGINX setups (multi-VMs, multi-replicas) ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-27+3.0.3-1.el7.ngx.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-27+3.0.3-1.el8.ngx.el8.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_27+3.0.3-1~buster_amd64.deb ##### Debian 11 - app-protect-dos_27+3.0.3-1~bullseye_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_27+3.0.3-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_27+3.0.3-1~focal_amd64.deb ##### Alpine 3.15 - app-protect-dos-27.3.0.3-r1.apk #### NGINX Plus - NGINX Plus R27 ### Important Notes - L4 (eBPF) mitigation helps mitigate volumetric attacks by slowing down the opening of TCP connections by the attackers. It is recommended to deploy F5 DoS for NGINX with L4 (eBPF) mitigation at the perimeter network or behind L3 load balancer. Installing F5 DoS for NGINX with L4 (eBPF) mitigation behind L4/L7 load balancer may result in the load balancer's starvation during an attack. - If F5 WAF for NGINX is installed, app protect should be disabled for the location of DoS Live Activity Monitoring API. For example: ```shell location /api { app_protect_enable off; app_protect_dos_api; } ``` - Misconfiguration of `app_protect_dos_monitor` potentially can cause a false attack declaration. Port configuration should correspond to the port the server listens to. For example: ```shell server { listen 8080; location / { app_protect_dos_monitor "myservice.com:8080"; } } ``` - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "Access Log Request Mechanism" url: https://docs.nginx.com/nginx-app-protect-dos/monitoring/access-log/index.md description: Learn about the F5 DoS for NGINX Request Log Mechanism. -------------------------------------------------------------------------------- ## Access Logs Access Log is NGINX’s request log mechanism. It is controlled by the following two directives.
### log_format This directive determines the format of the log messages using predefined variables. App Protect DoS will enrich this set of variables with several security log attributes that are available to be included in the `log_format`. If `log_format` is not specified then the built-in format `combined` is used but, because that format does not include the extended App Protect DoS variables, this directive must be used when the user wants to add App Protect DoS information to the log. ### access_log This directive determines the destination of the `access_log` and the name of the format according to the official [F5 NGINX documentation](https://docs.nginx.com). For example: `access_log /var/log/nginx/access.log log_dos`; (`log_dos` is predefined in the log_format directive). ## App Protect Variables for Access Log These are the variables added to Access Log. They are a subset of the Security log attributes. The Security log names are prefixed with `$app_protect_dos`. |Name| Meaning |Comment| |--- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------ | |`$app_protect_dos_outcome`| One of:
**Allow**: request was sent to origin server
**Redirect**: http redirection
**Challenge**: JS challenge
**Block**: blocked request || |`$app_protect_dos_outcome reason`| One of:
**Allow**: Request not mitigated, passed DoS flow successfully.
**Allowlist**: Request not mitigated because it is on the allowlist.
**Bypass**: Request not mitigated due to internal failure.
**Bad_Actor:**: Request mitigated as a bad actor.
**Signature**: Request mitigated as a matched DoS attack signature.
**Global_Rate**: Request mitigated as exceeding the calculated global request rate.
**Slow_Body**: Request mitigated due to being a slow request. |Combine `MITIGATED_BY_GLOBAL_RATE` with global rate value (in RPS) for example `Global_Rate, value=152,` | |`$app_protect_dos_tls_fp`| TLS Fingerprint - a value which identifies the sender |Applicable only in TLS (SSL) traffic| |`$app_protect_dos_policy_name`| The name of the policy that enforced the request || |`$app_protect_dos_vs_name`| The name of the protected object || |`$app_protect_dos_version`| The App Protect DoS version string:
major.minor.build format. |Does not include the F5 NGINX plus version (e.g. R21). The latter is available in `$version` variable.| **Note:** Many of the other Security log attributes that are not included here have exact or similar parallels among the NGINX variables also available for access log. For example, `$request` is parallel to the `request` security log attribute. See the full list of [NGINX variables](https://nginx.org/en/docs/http/ngx_http_log_module.html). ## Logging Rate Limit - mandatory configuration During a DoS attack, there is a large quantity of incoming requests which can flood the Access Log. The rate of the access log's entries can be limited in order to avoid this flood. NGINX logs all the requests during peacetime and logs up to 10 entries per second for each outcome reason during attack time. In worst case it can be 50 requests per second under attack. Two things should be configured in the `nginx conf` file: 1. Create a variable called `loggable` using NGINX's `set` directive and give it any value (string or numerical).
Note that the scope of the `set` directive is **server** or **location** block.
For example: **set $loggable '1'**; 2. Add the string **"if=$loggable"** to the **access_log** directive's argument. For example: access_log /var/log/nginx/access.log custom **if=$loggable**; ## Example ```nginx http { log_format security_dos 'request_time=$request_time client_ip=$remote_addr,' 'request="$request", status=$status,' 'dos_policy=$app_protect_dos_policy_name, dos_protected_object=app_protect_dos_vs_name' 'dos_action=$app_protect_dos_outcome, dos_action_reason=$app_protect_dos_outcome_reason'; server { location / { set $loggable 1; access_log /var/log/nginx/access.log security_dos if=$loggable;; ... } } } ``` -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 2.4" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-2.4/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v2.4. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 2.4 June 28, 2022 ### New Features - [Support for Debian 11](learn-about-deployment.md#debian-10--debian-11-installation) - Accelerated bad actors detection when the signatures mitigation is disabled. ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-27+2.4.1-1.el7.ngx.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-27+2.4.1-1.el8.ngx.el8.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_27+2.4.1-1~buster_amd64.deb ##### Debian 11 - app-protect-dos_27+2.4.1-1~bullseye_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_27+2.4.1-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_27+2.4.1-1~focal_amd64.deb ##### Alpine 3.15 - app-protect-dos-27.2.4.1-r1.apk #### NGINX Plus - NGINX Plus R27 ### Resolved Issues - Honoring allow/deny directives for restricting access to the F5 DoS for NGINX Live Activity Monitoring API location. ### Important Notes - If F5 WAF for NGINX is installed, app protect should be disabled for the location of DoS Live Activity Monitoring API For example: ```shell location /api { app_protect_enable off; app_protect_dos_api; } ``` - Misconfiguration of `app_protect_dos_monitor` potentially can cause a false attack declaration. Port configuration should correspond to the port the server listens to. For example: ```shell server { listen 8080; location / { app_protect_dos_monitor "myservice.com:8080"; } } ``` - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 2.3" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-2.3/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v2.3. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 2.3 May 9, 2022 ### New Features - [Support for Alpine 3.15 deployment](learn-about-deployment.md#alpine-315-installation) - [DoS Live Activity Monitoring](/nap-dos/monitoring/live-activity-monitoring.md) - [New Arbitrator FQDN/IP directive](learn-about-directives-and-policy.md#arbitrator-fqdn-directive-app_protect_dos_arb_fqdn) ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-26+2.3.46-1.el7.ngx.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-26+2.3.46-1.el8.ngx.el8.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_26+2.3.46-1~buster_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_26+2.3.46-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_26+2.3.46-1~focal_amd64.deb ##### Alpine 3.15 - app-protect-dos-26.2.3.48-r1.apk #### NGINX Plus - NGINX Plus R26 ### Resolved Issues - Improved Security logger configuration parsing: Detects invalid JSON structure, generates an appropriate message if field/value is not valid, doesn't stop on the first failure. ### Important Notes - Misconfiguration of `app_protect_dos_monitor` potentially can cause a false attack declaration. Port configuration should correspond to the port the server listens to. For example: ```shell server { listen 8080; server_name myservice.com; location / { app_protect_dos_monitor "myservice.com:8080/"; } } ``` Please note that the above syntax for the Monitor directive is old but still supported. The new syntax for the Monitor directive is as stated below, where **protocol** and **timeout** arguments are optional and default. Refer to [Learn about Directives and Policy](/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/#monitor-directive-app_protect_dos_monitor) for more details on Monitor directive new syntax. For example: `app_protect_dos_monitor uri=myservice.com:8080/ protocol=http1 timeout=5;`

- `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7/UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 2.2" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-2.2/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v2.2. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 2.2 February 15, 2022 In this release, support for F5 DoS for NGINX is added to NGINX Plus R26. ### New Features - **Improve Signature Lifecycle** ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-26+2.2.20-1.el7.ngx.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-26+2.2.20-1.el8.ngx.el8.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_26+2.2.20-1~buster_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_26+2.2.20-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_26+2.2.20-1~focal_amd64.deb #### NGINX Plus - NGINX Plus R26 ### Resolved Issues - Monitor requests are sent in new connections. It improves monitoring health capability and allows better detection of slow POST attacks. - Adaptive memory allocation by adminstall in order to support maximum cores. The amount of allocated memory for F5 DoS for NGINX is controlled by an argument of adminstall. The default value is 80 MB. For certain deployments, this amount of memory size is not enough. The default memory size is adaptive now. If the Virtual Machine (VM) has a big number of CPU cores, then we can also increase the amount of memory. For the case of up to 4 CPU cores, the allocated memory is 80MB, for more than 4 CPU cores, the allocated memory will be calculated as 80MB + CPUs * 2.5MB ### Important Notes - Misconfiguration of `app_protect_dos_monitor` can cause a false attack declaration. Port configuration should correspond to the port the server listens to. For example: ```shell server { listen 8080; server_name myservice.com; location / { app_protect_dos_monitor "myservice.com:8080/"; } } ``` - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "Operation Log" url: https://docs.nginx.com/nginx-app-protect-dos/monitoring/operation-log/index.md description: Learn about the F5 DoS for NGINX Operation Log. -------------------------------------------------------------------------------- ## Overview The operation logs consists of system operational and health events. The events are sent to the NGINX error log and are distinguished by the `APP_PROTECT_DOS` prefix followed by JSON body. The log level depends on the event: success is usually `notice` while failure is `error`. The timestamp is inherent in the error log. ## Events |Event Type|Level|Meaning| |--------- |-----|------ | |Configuration Error |error |There were errors in the [directives](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md) in the `nginx.conf` file.
Configuration error event is produced when one of F5 DoS for NGINX directives is supplied with an incorrect data. An additional information will be added to the message, describing what was incorrect. NGINX will run with default values for this directive.
Please note that if the directive supplied with an incorrect number of arguments then NGINX will issue an error and NGINX will not run. It is a generic NGINX behavior.| ```json { "event": "configuration_error", "error_message": "unknown argument", "line_number": 58 } ``` |Event Type|Level|Meaning| |--------- |-----|------ | |Configuration Load Failure |error |There was an error in one of the configuration files: file not found, failed to parse.| ```json { "event": "configuration_load_failure", "error_message": "Failed to load Policy '/etc/app_protect_dos/BADOSDefaultPolicy.json' : Fail parse JSON Policy: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"xxxx\\nhdjk\\n\\n555\\n\") \n.\n", "error_line_number": 58 } ``` |Event Type|Level|Meaning| |--------- |-----|------ | |Configuration Load Success | notice |The `APP_PROTECT_DOS` configuration process ended successfully: all policies, log configuration and global settings were loaded to F5 DoS for NGINX and all traffic will be handled by this configuration. The `error_message` contains warnings. This event is also generated on the initial configuration (when NGINX Plus starts).| ```json { "event": "configuration_load_success", "software_version": "x.x.x.x.x", "error_message": "warning if exists..." } ``` |Event Type|Level|Meaning| |--------- |-----|------ | |Shared Memory Failure |error |A worker attempted to connect to shared memory but the operation failed. One time error (per worker) - upon switch from `init` or `operational` mode to `failure`. The `mode` should be `failure`.| ```json { "event": "shared_memory_failure", "worker_pid": 4928, "mode": "failure", "mode_changed": true } ``` |Event Type|Level|Meaning| |--------- |-----|------ | |Shared Memory Connected |notice |A worker successfully connected to shared memory.
One time message (per worker) - upon switch from init or failure mode to `operational`.
The `mode` attribute should be `operational`, unless there is an ongoing problem.| ```json { "event": "shared_memory_connected", "worker_pid": 4928, "mode": "operational", "mode_changed": true } ``` |Event Type|Level|Meaning| |--------- |-----|------ | | eBPF Failure | error | A worker attempted to connect to eBPF maps but the operation failed. One time error (per worker) - upon switch from `init` or `operational` mode to `failure`. The mode should be `failure`.| ```json { "event": "ebpf_failure", "worker_pid": 4928, "mode": "failure", "mode_changed": true } ``` |Event Type|Level|Meaning| |--------- |-----|------ | | eBPF Connected | notice | A worker successfully connected to eBPF maps.
One time message (per worker) - upon switch from `init` or `failure` mode to `operational`.
The `mode` attribute should be `operational`, unless there is an ongoing problem. | ```json { "event": "ebpf_connected", "worker_pid": 4928, "mode": "operational", "mode_changed": true } ``` -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 2.1" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-2.1/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v2.1. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 2.1 December 29, 2021 ### New Features - [Support for RHEL (7.4.x and above) Virtual Machine (VM) deployment](learn-about-deployment.md#rhel-74-installation) - [Support for RHEL 8 and UBI 8](learn-about-deployment.md#rhel-8-installation) - [GRPC and HTTP/2 protection support for Centos (7.4.x and above) and RHEL (7.4.x and above)](/nap-dos/deployment-guide/learn-about-deployment.md) ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / RHEL 7.4+ / UBI7 - app-protect-dos-25+2.1.8-1.el7.ngx.el7.ngx.x86_64.rpm ##### RHEL 8 / UBI8 - app-protect-dos-25+2.1.8-1.el8.ngx.el8.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_25+2.1.8-1~buster_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_25+2.1.8-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_25+2.1.8-1~focal_amd64.deb #### NGINX Plus - NGINX Plus R25 ### Resolved Issues - The `app_protect_dos_name` directive is not inherited by the inner blocks, causing to have more VSs than expected. - Signature should not be created if good and bad actor use the same type of traffic. - When there's a clear anomaly on the User-Agent header signal, the signature doesn't include it. - HTTP Method signal is named incorrectly in signatures. ### Important Notes - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 and RHEL 7 / UBI 7 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - The recommended option of running NGINX Plus in a Docker Container is with the `daemon off` flag. It's mandatory for UBI 8. -------------------------------------------------------------------------------- title: "Troubleshooting Guide" url: https://docs.nginx.com/nginx-app-protect-dos/troubleshooting/how-to-troubleshoot/index.md description: Learn about the F5 DoS for NGINX Troubleshooting Guide. -------------------------------------------------------------------------------- ## Overview This Troubleshooting Guide is intended to provide guidance to customers in the detection and correction of programming issues in F5 DoS for NGINX. It may also be useful to IT. ## Resolving Known Problems ### Configuration |Problem|Solution| |-------|--------| | NGINX is not running (ps -aux)

Reloading NGINX fails| Check the error log at `/var/log/nginx/error.log`.
Fix the problem and re-run NGINX.| | No original source IP in logs|1. XFF is not configured (or not configured correctly)
2. External Load Balancer doesn't forward XFF | | F5 DoS for NGINX functionality is not as expected| F5 DoS for NGINX has several logs which can be used for troubleshooting.
Usually, it is best to look for any warning or error messages within the logs.
Refer to [Logs Overview](/nap-dos/monitoring/types-of-logs.md)| | `Too many open files` error message | Increase number of file descriptors.
For example: `worker_rlimit_nofile 65535;` in the main context of `nginx.conf` file.
Refer to [worker_rlimit_nofile directive](https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile) | | `setrlimit ... failed (Permission denied)` error message | Increase the limit using the following command as the root user:
`setsebool -P httpd_setrlimit 1;`
Refer to [Issue 4: Too many files are open Error](https://www.f5.com/company/blog/nginx/using-nginx-plus-with-selinux/) | | More protected objects than expected | The `app_protect_dos_enable` directive is inherited by all server and location blocks beneath it, each block will be a protected object.
Consider moving this directive from outer to inner block.
Refer to: [F5 DoS for NGINX - Directives and Policy](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md) | | `No DOS protection for ngx_worker at idx X` warning message | There are more nginx processes than allowed.
Either decrease the number of nginx processes (ngx_processes directive in `nginx.conf` file) or increase the number of supported workers for F5 DoS for NGINX using the flag `--max-workers NUM` for `/usr/bin/adminstall`. | | `unknown directive 'app_protect_dos_xxx'` error message | App Protect DOS module is not loaded. Add this line to the main (global) context of nginx.conf:
`load_module "/etc/nginx/modules/ngx_http_app_protect_dos_module.so";` | | NGINX struggles handling a high rate of incoming connections | Linux machine should be tuned for optimal performance.
Refer to [Tuning NGINX for Performance](https://www.nginx.com/blog/tuning-nginx/) | | Error in `adminstall` process, such as `Failed to allocate` | Insufficient memory to allocate all the required resources.
Increase the `--memory` size or decrease the number of nginx workers (`--max_workers`) if not all of them are going to be in use.
Use the `--help` flag for more info. | ### ELK issues ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for [F5 DoS for NGINX GitHub repo](https://github.com/f5devcentral/nap-dos-elk-dashboards). ### SELinux Configure SELinux to allow F5 DoS for NGINX. The configuration steps are found in the [SELinux configuration](/nap-dos/deployment-guide/learn-about-deployment.md#selinux-configuration) section of the deployment guide. If SELinux still denies access to something, it means that one of more security exceptions should be enabled. The following steps describe how to find the problematic exception and enable it. 1. Temporarily add the `httpd_t` domain to the permissive list (this will completely enable all the fields).
In this configuration SELinux will not deny anything related to NGINX as NGINX is labeled with the `httpd_t` context.
```shell semanage permissive -a httpd_t ``` 2. Repeat the scenario which made SELinux deny and see that it now works. 3. In permissive mode, security exceptions are logged to the default Linux audit log. Due to the previous step, the permitted exception will be logged.
The log can be found in `/var/log/audit/audit.log`. 4. The following command will parse the audit log and build a SELinux command that will permit all the exceptions found in the log: ```shell grep nginx /var/log/audit/audit.log | audit2allow -m nginx ``` 5. Compare the generated output to the nginx.te file mentioned in the deployment guide. Add all the missing commands to the nginx.te file and repeat the SELinux configuration mentioned in the deployment guide. 6. Delete the `httpd_t` domain from the permissive list: ```shell semanage permissive -d httpd_t ``` For more information about how to use NGINX Plus with SELinux - check our [blog](https://www.f5.com/company/blog/nginx/using-nginx-plus-with-selinux/) ### Send Logs to Support If there are any problems, collect the troubleshooting information in a tarball and send it to your customer support engineer. 1. Get package version: a. Get F5 DoS for NGINX version:
```shell /usr/bin/admd -v > package_versions.txt ``` b. Get packages version:
For CentOS/RHEL:
```shell rpm -qa nginx-plus* app-protect* >> package_versions.txt ``` For Debian/Ubuntu:
```shell apt list --installed | grep -E 'nginx-plus|app-protect' >> package_versions.txt ``` c. Get OS version:
```shell cat /etc/os-release > system_version.txt && uname -r >> system_version.txt && cat /proc/version >> system_version.txt ``` d. Get F5 DoS for NGINX shared memory dump:
```shell admd -c > napd_shmem.txt ``` e. Get Linux shared memory dump:
```shell ipcs -m > linux_shmem.txt ``` 2. Create a list of files for tarball:
a. Create a file using your favorite editor (i.e VI editor)
```shell vi logs.txt ``` b. Insert the following content into the file created above:
```shell package_versions.txt system_version.txt napd_shmem.txt linux_shmem.txt /var/log/adm/* /var/run/adm/* /var/log/nginx/* ``` c. Add the path of your NGINX configuration files including all references, for example:
```shell /etc/nginx/nginx.conf /etc/nginx/conf.d/* ``` d. Add all policies and log file configuration, for example:
```shell /etc/app_protect_dos/* ``` 3. Create the tarball: ```shell tar cvfz logs.tgz `cat logs.txt` ``` 4. Send `logs.tgz` to your customer support. -------------------------------------------------------------------------------- title: "F5 DoS for NGINX 2.0" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-2.0/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v2.0. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 2.0 October 20, 2021 ### New Features #### [Protection of gRPC services against application layer Denial of Service attacks](/nap-dos/deployment-guide/learn-about-deployment.md) - **Message flood** Attacker supplies multiple gRPC requests that exceed the service capacity. - **Concurrent large requests** Attacker supplies a number of concurrent large requests that exceed the server capacity of concurrent requests. - **Slow gRPC POST** Attacker supplies a number of concurrent slow POST gRPC requests that exceed the server capacity of concurrent requests. - **HTTP/2 attack on gRPC service** Attacker runs typical DoS HTTP/2 attacks: HTTP flood and slow attacks on gRPC service. #### [Protection of HTTP/2 services against application layer Denial of Service attacks](/nap-dos/deployment-guide/learn-about-deployment.md) ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ / UBI7 - app-protect-dos-25+2.0.1-1.el7.ngx.el7.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_25+2.0.1-1~buster_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_25+2.0.1-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_25+2.0.1-1~focal_amd64.deb #### NGINX Plus - NGINX Plus R25 ### Resolved Issues - Security log keeps working on removed Protected Objects. - Monitoring requests show up in the access log. - `app_protect_dos_name` longer than 32 characters creates a garbage name in the logs. - Created protected objects for the not configured contexts. - Wrong reporting of attack status with arbitrator. - Wrong `impact_rps` value in **Bad actor expired** log message. - Rate limit in Access Log should be optional. ### Important Notes - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - gRPC and HTTP/2 protection are available only on Debian 10, Ubuntu 18.04 and Ubuntu 20.04 platforms. For the rest of the platforms, F5 DoS for NGINX does not protect gRPC and HTTP/2 services. The traffic is bypassed. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - New optional configuration parameters of the directive `app_protect_dos_monitor` to support gRPC and HTTP/2 protocols. - Added new fields in Security Log: - `baseline_dps` (datagrams per second) instead of `baseline_tps`, `incoming_datagrams`
- `successful_responses` instead of `successful_transactions`
- `unsuccessful_requests` instead of `unsuccessful_requests_count`. - In the case of an upgrade from the previous `app-protect-dos` version, it's necessary to remove the old `nginx-plus` and install the new `app-protect-dos` that will install a correspondent version of `nginx-plus` as described in the [F5 DoS for NGINX Deployment Guide](/nap-dos/deployment-guide/learn-about-deployment.md). -------------------------------------------------------------------------------- title: "F5 DoS for NGINX Arbitrator 1.1.0" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-1.1.0/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX Arbitrator v1.1.0. ## Arbitrator Service Release 1.1.0 December 1, 2021 This release is focused on security and stability. ### New Features - Improve security by enabling the arbitrator to work as a non-root user. - Remove operating system dependencies to work as a native service utilizing golang. ### Resolved Issues - Special characters like a slash inside the protected object name prevented Arbitrator to save the state file. ### Important Notes - The current release upgrades Arbitrator service only. This change is agnostic to F5 DoS for NGINX functionalities. - `proxy_request_buffering` off is not supported. - gRPC and HTTP/2 protection require active monitoring of the protected service. The directive `app_protect_dos_monitor` is mandatory for these use cases, otherwise, the attack will not be detected. - gRPC and HTTP/2 protection are available only on Debian 10, Ubuntu 18.04, and Ubuntu 20.04 platforms. For the rest of the platforms, F5 DoS for NGINX does not protect gRPC and HTTP/2 services. The traffic is bypassed. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. - New optional configuration parameters of the directive `app_protect_dos_monitor` to support gRPC and HTTP/2 protocols. - Added new fields in Security Log: - `baseline_dps` (datagrams per second) instead of `baseline_tps`, `incoming_datagrams`
- `successful_responses` instead of `successful_transactions`
- `unsuccessful_requests` instead of `unsuccessful_requests_count`. - In the case of an upgrade from the previous `app-protect-dos` version, it's necessary to remove the old `nginx-plus` and install the new `app-protect-dos` that will install a correspondent version of `nginx-plus` as described in the [F5 DoS for NGINX Deployment Guide](/nap-dos/deployment-guide/learn-about-deployment.md). -------------------------------------------------------------------------------- title: "NGINX App Protect 1.0" url: https://docs.nginx.com/nginx-app-protect-dos/releases/about-1.0/index.md -------------------------------------------------------------------------------- Here you can find the release information for F5 DoS for NGINX v1. F5 DoS for NGINX provides behavioral protection against Denial of Service (DoS) for your web applications. ## Release 1.0 July 6, 2021 ### New Features #### Protection against application layer Denial of Service attacks - GET and POST flood
- Slowloris, Slowread, Slowpost
- Distributed variations of attacks (see above)
- Challenge Collapsar (CC) attack/random URIs
#### False-positive management mechanisms - HTTP Redirection - Client-side validation - TLS fingerprinting #### Use Cases #### Application Types - Traditional HTML-based web applications - XML-based web services - REST APIs (JSON) #### Deployment Options - Kubernetes Per-pod proxy - Kubernetes Per-service proxy - API Gateway - Traditional edge proxy ### Supported Packages #### App Protect DoS ##### CentOS 7.4+ - app-protect-dos-24+1.69.6-1.el7.ngx.el7.ngx.x86_64.rpm ##### Debian 10 - app-protect-dos_24+1.69.6-1~buster_amd64.deb ##### Ubuntu 18.04 - app-protect-dos_24+1.69.6-1~bionic_amd64.deb ##### Ubuntu 20.04 - app-protect-dos_24+1.69.6-1~focal_amd64.deb #### NGINX Plus - NGINX Plus R24 ### Known Issues - `proxy_request_buffering off` is not supported. - F5 DoS for NGINX does not protect `grpc` and `http2` services. The traffic is bypassed. - [TLS fingerprint](/nap-dos/directives-and-policy/learn-about-directives-and-policy.md#policy-directive-app_protect_dos_policy_file) feature is not used in CentOS 7.4 due to the old OpenSSL version. The required OpenSSL version is 1.1.1 or higher. - Slow POST attack always mitigates with block action while other types of attacks can also be mitigated with redirection or JS challenges. -------------------------------------------------------------------------------- title: "F5 NGINX Gateway Fabric" url: https://docs.nginx.com/nginx-gateway-fabric/index.md -------------------------------------------------------------------------------- ## About NGINX Gateway Fabric provides an implementation of the [Gateway API](https://gateway-api.sigs.k8s.io/) using [NGINX](https://nginx.org/) as the data plane. The goal of the project is to implement the core Gateway APIs needed to configure an HTTP or TCP/UDP load balancer, reverse proxy, or API gateway for Kubernetes applications. ## Featured content ### [Get started](/nginx-gateway-fabric/get-started) Use kind to quickly deploy and test a NGINX Gateway Fabric cluster. ### [Deploy NGINX Gateway Fabric](/nginx-gateway-fabric/install) Choose how to deploy NGINX Gateway Fabric in production. ### [Gateway API compatibility](/nginx-gateway-fabric/overview/gateway-api-compatibility/) View how much of the Gateway API NGINX Gateway Fabric supports. ## Design NGINX Gateway Fabric separates the control plane and data plane into distinct deployments. The control plane interacts with the Kubernetes API, watching for Gateway API resources. When a new Gateway resource is provisioned, it dynamically creates and manages a corresponding NGINX data plane Deployment and Service. Each NGINX data plane pod consists of an NGINX container integrated with [NGINX Agent](https://github.com/nginx/agent). The control plane translates Gateway API resources into NGINX configurations and sends these configurations to the agent to ensure consistent traffic management. This design enables centralized management of multiple Gateways while ensuring that each NGINX instance stays aligned with the cluster’s current configuration. For more information, see the [Gateway architecture](/ngf/overview/gateway-architecture.md) topic. ## More information ### [Changelog](/nginx-gateway-fabric/changelog) Review the changes from the latest releases. ### [Technical specifications](/nginx-gateway-fabric/overview/technical-specifications) Check which versions of NGINX Gateway Fabric match the API. ### [Routing traffic to applications](/nginx-gateway-fabric/traffic-management/basic-routing) Create simple rules for directing network traffic with HTTPRoute resources. ### [Secure traffic using Let's Encrypt](/nginx-gateway-fabric/traffic-security/integrate-cert-manager) Implement HTTPS with Let's Encrypt to secure client-server communication. -------------------------------------------------------------------------------- title: "Add certificates for secure authentication" url: https://docs.nginx.com/nginx-gateway-fabric/install/secure-certificates/index.md -------------------------------------------------------------------------------- By default, NGINX Gateway Fabric installs self-signed certificates to secure the connection between the NGINX Gateway Fabric control plane and the NGINX data plane pods. These certificates are created by a `cert-generator` job when NGINX Gateway Fabric is first installed. However, because these certificates are self-signed and will expire after 3 years, we recommend a solution such as [cert-manager](https://cert-manager.io) to create and manage these certificates in a production environment. This guide will step through how to install and use `cert-manager` to secure this connection. **Note:** These steps should be completed before you install NGINX Gateway Fabric. --- ## Before you begin To complete this guide, you will need the following prerequisites: - Administrator access to a Kubernetes cluster. - [Helm](https://helm.sh) and [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) must be installed locally. ## Install Gateway API CRDs ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl apply -f - ``` ## Install cert-manager Add the Helm repository: ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` Install cert-manager: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` This also enables Gateway API features for cert-manager, which can be useful for [securing your workload traffic](/ngf/traffic-security/integrate-cert-manager.md). ## Create the CA issuer The first step is to create the CA (certificate authority) issuer. **Note:** This example uses a self-signed Issuer, which should not be used in production environments. For production environments, you should use a real [CA issuer](https://cert-manager.io/docs/configuration/ca/). Create the namespace: ```shell kubectl create namespace nginx-gateway ``` ```yaml kubectl apply -f - <-nginx-gateway-fabric..svc`. The default Helm release name used in our installation docs is `ngf`, and the default namespace is `nginx-gateway`, so the `dnsName` should be `ngf-nginx-gateway-fabric.nginx-gateway.svc`. #### Manifests The full service name is of the format: `..svc`. By default, the base service name is `nginx-gateway`, and the namespace is `nginx-gateway`, so the `dnsName` should be `nginx-gateway.nginx-gateway.svc`. ```yaml kubectl apply -f - <Packages:

gateway.nginx.org/v1alpha1

Package v1alpha1 contains API Schema definitions for the gateway.nginx.org API group.

Resource Types:

AuthenticationFilter

AuthenticationFilter configures request authentication and is referenced by HTTPRoute and GRPCRoute filters using ExtensionRef.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		AuthenticationFilter
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
AuthenticationFilterSpec

Spec defines the desired state of the AuthenticationFilter.



basic
BasicAuth 		(Optional)

Basic configures HTTP Basic Authentication.
oidc
OIDCAuth 		(Optional)

OIDC configures OpenID Connect Authentication (NGINX Plus).
jwt
JWTAuth 		(Optional)

JWT configures JSON Web Token authentication (NGINX Plus).
type
AuthType

Type selects the authentication mechanism.
status
AuthenticationFilterStatus

Status defines the state of the AuthenticationFilter.

ClientSettingsPolicy

ClientSettingsPolicy is an Inherited Attached Policy. It provides a way to configure the behavior of the connection between the client and NGINX Gateway Fabric.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		ClientSettingsPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ClientSettingsPolicySpec

Spec defines the desired state of the ClientSettingsPolicy.



body
ClientBody 		(Optional)

Body defines the client request body settings.
keepAlive
ClientKeepAlive 		(Optional)

KeepAlive defines the keep-alive settings.
targetRef
sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRef identifies an API object to apply the policy to. Object must be in the same namespace as the policy. Support: Gateway, HTTPRoute, GRPCRoute.
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the ClientSettingsPolicy.

ExternalLoadBalancer

ExternalLoadBalancer configures an external load balancer that fronts a Gateway. It references a Gateway through TargetRefs. NGINX Gateway Fabric provisions the external load balancer integration for the Gateway’s data plane Service.

ExternalLoadBalancer maps one-to-one to a Gateway: a Gateway yields exactly one data plane Service, so it is fronted by exactly one ExternalLoadBalancer. When more than one ExternalLoadBalancer references the same Gateway, the oldest is accepted and the others are rejected with Accepted=False.

A resource configures exactly one external load balancer backend. The gatewayLink backend integrates F5 BIG-IP through F5 CIS.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		ExternalLoadBalancer
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ExternalLoadBalancerSpec

Spec defines the desired state of the ExternalLoadBalancer.



gatewayLink
GatewayLinkConfig 		(Optional)

GatewayLink configures F5 BIG-IP as the external load balancer using F5 Container Ingress Services. It is the first supported backend. Additional backend types may be added as sibling fields in the future.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies the Gateways this external load balancer applies to. Each object must be in the same namespace as the ExternalLoadBalancer resource. Exactly one Gateway is supported for now. Support: Gateway.
status
ExternalLoadBalancerStatus

Status defines the state of the ExternalLoadBalancer.

NginxGateway

NginxGateway represents the dynamic configuration for an NGINX Gateway Fabric control plane.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		NginxGateway
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
NginxGatewaySpec

NginxGatewaySpec defines the desired state of the NginxGateway.



logging
Logging 		(Optional)

Logging defines logging related settings for the control plane.
status
NginxGatewayStatus

NginxGatewayStatus defines the state of the NginxGateway.

PayloadProcessor

PayloadProcessor is an Inherited Attached Policy. It enables declarative processing of HTTP request and response payload content by attaching to a Gateway or HTTPRoute.

note: this CRD is based on the proposed “PayloadProcessor” policy from the Gateway API. Once the upstream API is finalized, migration to the official API will be considered.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		PayloadProcessor
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
PayloadProcessorSpec

Spec defines the desired state of the PayloadProcessor.



targetRef
sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRef identifies the Gateway or HTTPRoute this policy applies to. Objects must be in the same namespace as the policy. Follows the standard policy attachment pattern (GEP-713).

Support: Gateway, HTTPRoute
processors
[]PayloadProcessorEntry

Processors is an ordered list of processing steps to be applied to the request and response payloads. It is currently limited to a single processor (MaxItems=1); the list form is reserved for future multi-processor pipelines.
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the PayloadProcessor.

ProxySettingsPolicy

ProxySettingsPolicy is an Inherited Attached Policy. It provides a way to configure the behavior of the connection between NGINX Gateway Fabric and the upstream applications (backends).

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		ProxySettingsPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ProxySettingsPolicySpec

Spec defines the desired state of the ProxySettingsPolicy.



buffering
ProxyBuffering 		(Optional)

Buffering configures the buffering of responses from the proxied server.
timeout
ProxyTimeout 		(Optional)

Timeout configures timeouts for the connection to the proxied server.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies the API object(s) to apply the policy to. Objects must be in the same namespace as the policy. Support: Gateway, HTTPRoute, GRPCRoute
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the ProxySettingsPolicy.

RateLimitPolicy

RateLimitPolicy is an Inherited Attached Policy. It provides a way to set local rate limiting rules in NGINX.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		RateLimitPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
RateLimitPolicySpec

Spec defines the desired state of the RateLimitPolicy.



rateLimit
RateLimit 		(Optional)

RateLimit defines the Rate Limit settings.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to. Objects must be in the same namespace as the policy.

Support: Gateway, HTTPRoute, GRPCRoute
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the RateLimitPolicy.

SnippetsFilter

SnippetsFilter is a filter that allows inserting NGINX configuration into the generated NGINX config for HTTPRoute and GRPCRoute resources.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		SnippetsFilter
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
SnippetsFilterSpec

Spec defines the desired state of the SnippetsFilter.



snippets
[]Snippet

Snippets is a list of NGINX configuration snippets. There can only be one snippet per context. Allowed contexts: main, http, http.server, http.server.location.
status
SnippetsFilterStatus

Status defines the state of the SnippetsFilter.

SnippetsPolicy

SnippetsPolicy provides a way to inject NGINX snippets into the configuration on Gateway level.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		SnippetsPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
SnippetsPolicySpec

Spec defines the desired state of the SnippetsPolicy.



targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to.
snippets
[]Snippet 		(Optional)

Snippets is a list of snippets to be injected into the NGINX configuration.
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the current state of the SnippetsPolicy.

UpstreamSettingsPolicy

UpstreamSettingsPolicy is a Direct Attached Policy. It provides a way to configure the behavior of the connection between NGINX and the upstream applications.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		UpstreamSettingsPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
UpstreamSettingsPolicySpec

Spec defines the desired state of the UpstreamSettingsPolicy.



zoneSize
Size 		(Optional)

ZoneSize is the size of the shared memory zone used by the upstream. This memory zone is used to share the upstream configuration between nginx worker processes. The more servers that an upstream has, the larger memory zone is required. Default: OSS: 512k, Plus: 1m. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone
keepAlive
UpstreamKeepAlive 		(Optional)

KeepAlive defines the keep-alive settings.
loadBalancingMethod
LoadBalancingType 		(Optional)

LoadBalancingMethod specifies the load balancing algorithm to be used for the upstream. If not specified, NGINX Gateway Fabric defaults to random two least_conn, which differs from the standard NGINX default round-robin.
hashMethodKey
HashMethodKey 		(Optional)

HashMethodKey defines the key used for hash-based load balancing methods. This field is required when LoadBalancingMethod is set to hash or hash consistent.
useClusterIP
bool 		(Optional)

UseClusterIP configures NGINX to route to the Service ClusterIP and port instead of individual Pod IPs. When enabled, NGINX will target a single upstream server corresponding to the Service’s ClusterIP, which is useful for service mesh compatibility and other Kubernetes controllers/operators that require traffic to traverse the Service VIP. This setting applies only when the target Service has a ClusterIP. For headless Services (ClusterIP: None) and ExternalName Services, normal endpoint resolution is used instead. This setting is also not applied to L4/stream upstreams. Defaults to false.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to. Objects must be in the same namespace as the policy. Support: Service

TargetRefs must be distinct. The name field must be unique for all targetRef entries in the UpstreamSettingsPolicy.
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the UpstreamSettingsPolicy.

WAFPolicy

WAFPolicy is an Inherited Attached Policy. It provides a way to configure F5 WAF for NGINX for Gateways and Routes by referencing compiled WAF policy bundles. Bundles can be fetched directly from an HTTP/HTTPS URL (type: HTTP), from an NGINX Instance Manager instance (type: NIM), from an F5 NGINX One Console instance (type: N1C), or from a Policy Lifecycle Manager’s S3-compatible storage (type: PLM).

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha1
kind
string 		WAFPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
WAFPolicySpec

Spec defines the desired state of the WAFPolicy.



targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to. Objects must be in the same namespace as the policy. All targets must be of the same Kind (all Gateways OR all HTTPRoutes OR all GRPCRoutes). Support: Gateway, HTTPRoute, GRPCRoute.
type
PolicySourceType

Type identifies the source type for the policy bundle. HTTP fetches directly from a URL; NIM uses the NGINX Instance Manager bundles API; N1C uses the F5 NGINX One Console security policies API; PLM references an APPolicy CRD managed by the Policy Lifecycle Manager.
policySource
PolicySource 		(Optional)

PolicySource holds all non-CRD bundle fetch configuration. Used for HTTP, NIM, and N1C policy types. Must not be set when type is PLM.
policyRef
PolicyRef 		(Optional)

PolicyRef holds all CRD-backed policy references. Used for the PLM policy type.
securityLogs
[]WAFSecurityLog 		(Optional)

SecurityLogs defines security logging configurations.
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the WAFPolicy.

APLogConfReference

(Appears on: LogRef)

APLogConfReference identifies an APLogConf resource managed by PLM.

Field Description
namespace
string 		(Optional)

Namespace is the namespace of the APLogConf resource. If not set, the namespace of the WAFPolicy is used. Cross-namespace references require a ReferenceGrant.
name
string

Name is the name of the APLogConf resource.

APPolicyReference

(Appears on: PolicyRef)

APPolicyReference identifies an APPolicy resource managed by PLM.

Field Description
namespace
string 		(Optional)

Namespace is the namespace of the APPolicy resource. If not set, the namespace of the WAFPolicy is used. Cross-namespace references require a ReferenceGrant.
name
string

Name is the name of the APPolicy resource.

AuthType (string alias)

(Appears on: AuthenticationFilterSpec)

AuthType defines the authentication mechanism.

Value Description

"Basic"

AuthTypeBasic is the HTTP Basic Authentication mechanism.

"JWT"

AuthTypeJWT is the JWT Authentication mechanism.

"OIDC"

AuthTypeOIDC is the OpenID Connect Authentication mechanism.

AuthenticationFilterConditionReason (string alias)

AuthenticationFilterConditionReason is a reason for an AuthenticationFilter condition type.

Value Description

"Accepted"

AuthenticationFilterConditionReasonAccepted is used with the Accepted condition type when the condition is true.

"Invalid"

AuthenticationFilterConditionReasonInvalid is used with the Accepted condition type when the filter is invalid.

AuthenticationFilterConditionType (string alias)

AuthenticationFilterConditionType is a type of condition associated with AuthenticationFilter.

Value Description

"Accepted"

AuthenticationFilterConditionTypeAccepted indicates that the AuthenticationFilter is accepted.

Possible reasons for this condition to be True: * Accepted

Possible reasons for this condition to be False: * Invalid.

AuthenticationFilterSpec

(Appears on: AuthenticationFilter)

AuthenticationFilterSpec defines the desired configuration.

Field Description
basic
BasicAuth 		(Optional)

Basic configures HTTP Basic Authentication.
oidc
OIDCAuth 		(Optional)

OIDC configures OpenID Connect Authentication (NGINX Plus).
jwt
JWTAuth 		(Optional)

JWT configures JSON Web Token authentication (NGINX Plus).
type
AuthType

Type selects the authentication mechanism.

AuthenticationFilterStatus

(Appears on: AuthenticationFilter)

AuthenticationFilterStatus defines the state of AuthenticationFilter.

Field Description
controllers
[]ControllerStatus

Controllers is a list of Gateway API controllers that processed the AuthenticationFilter and the status of the AuthenticationFilter with respect to each controller.

Authorization

(Appears on: JWTAuth, OIDCAuth)

Authorization specifies a set of required claim rules that a token’s claim must match to be authorized, given the require type defined.

Field Description
require
RequireType 		(Optional)

Require sets top level authorization requirement. When set to All, the requirements for all claims in a rule must be met. When set to Any, the requirements for any one claim in a rule must be met.
rules
[]Rule

Rules defines a list of claims and their specific authorization requirements.

BasicAuth

(Appears on: AuthenticationFilterSpec)

BasicAuth configures HTTP Basic Authentication.

Field Description
secretRef
LocalObjectReference

SecretRef references a Secret containing credentials in the same namespace.
realm
string

Realm used by NGINX auth_basic directive. https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html#auth_basic Also configures “realm=”” in WWW-Authenticate header in error page location.

BundleAuth

(Appears on: LogSource, PolicySource)

BundleAuth configures authentication for bundle fetching.

Field Description
secretRef
LocalObjectReference

SecretRef references a Kubernetes Secret in the same namespace as the WAFPolicy. The Secret may contain: - “username” and “password” fields for HTTP Basic Authentication - “token” field for Bearer Token Authentication (NIM) or APIToken Authentication (N1C)

BundlePolling

(Appears on: LogSource, PolicySource)

BundlePolling configures automatic re-fetching of a bundle.

Field Description
interval
Kubernetes meta/v1.Duration 		(Optional)

Interval is the period between poll cycles. Defaults to 5m when polling is enabled but no interval is set.
enabled
bool 		(Optional)

Enabled activates periodic re-fetching of the bundle. When true, NGF fetches the bundle on each interval and deploys it only if its checksum differs from the last successfully fetched version.

BundleValidation

(Appears on: LogSource, PolicySource)

BundleValidation configures integrity verification for a bundle. Exactly one of verifyChecksum or expectedChecksum may be set.

Field Description
expectedChecksum
string 		(Optional)

ExpectedChecksum is the expected SHA256 checksum of the bundle. If set, the downloaded bundle must match this checksum or it will be rejected. For N1C sources, the checksum reported by the N1C API is verified automatically; set this field only if you want to enforce an additional, independently known value.
verifyChecksum
bool 		(Optional)

VerifyChecksum enables automatic checksum verification by fetching a companion checksum file at .sha256 and comparing it against the downloaded bundle. Only supported when the policy source type is HTTP (policySource.httpSource or logSource.url); setting this for NIM or N1C sources is rejected at admission. Note: for N1C sources, bundle integrity is always verified automatically using the checksum returned by the N1C compile API — this field is not needed. Mutually exclusive with expectedChecksum.

Claim

(Appears on: Rule)

Claim describes the exact name/value pair of claims that must be matched.

Field Description
proxySetHeader
string

ProxySetHeader sets both the name and variable for proxy_set_header Example: For claim name sub for JWT auth

proxy_set_header X-JWT-Claim-Sub $jwt_claim_sub;
name
string

Name is the name of the claim within the token.
match
ClaimMatchType

Match sets the match type for the claim.
values
[]string

Values are the values within the claim. When more than one value is set, the claim must match any of these values.

ClaimMatchType (string alias)

(Appears on: Claim)

ClaimMatchType defines how claim values are parsed.

Value Description

"Exact"

ClaimMatchTypeExact treats claim values as their exact value.

"Regex"

ClaimMatchTypeRegex treats claim values as a regex value.

ClientBody

(Appears on: ClientSettingsPolicySpec)

ClientBody contains the settings for the client request body.

Field Description
maxSize
Size 		(Optional)

MaxSize sets the maximum allowed size of the client request body. If the size in a request exceeds the configured value, the 413 (Request Entity Too Large) error is returned to the client. Setting size to 0 disables checking of client request body size. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size.
bufferSize
Size 		(Optional)

BufferSize sets the buffer size for reading the client request body. If the request body is larger than the buffer, the whole body or only its part is written to a temporary file. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size.
timeout
Duration 		(Optional)

Timeout defines a timeout for reading client request body. The timeout is set only for a period between two successive read operations, not for the transmission of the whole request body. If a client does not transmit anything within this time, the request is terminated with the 408 (Request Time-out) error. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout.

ClientKeepAlive

(Appears on: ClientSettingsPolicySpec)

ClientKeepAlive defines the keep-alive settings for clients.

Field Description
requests
int32 		(Optional)

Requests sets the maximum number of requests that can be served through one keep-alive connection. After the maximum number of requests are made, the connection is closed. Closing connections periodically is necessary to free per-connection memory allocations. Therefore, using too high maximum number of requests is not recommended as it can lead to excessive memory usage. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests.
time
Duration 		(Optional)

Time defines the maximum time during which requests can be processed through one keep-alive connection. After this time is reached, the connection is closed following the subsequent request processing. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_time.
timeout
ClientKeepAliveTimeout 		(Optional)

Timeout defines the keep-alive timeouts for clients.
minTimeout
Duration 		(Optional)

MinTimeout defines the timeout for which the keep-alive client connection will not be closed on the server side for connection reuse or on graceful shutdown of worker processes. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_min_timeout.

ClientKeepAliveTimeout

(Appears on: ClientKeepAlive)

ClientKeepAliveTimeout defines the timeouts related to keep-alive client connections. Default: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout.

Field Description
server
Duration 		(Optional)

Server sets the timeout during which a keep-alive client connection will stay open on the server side. Setting this value to 0 disables keep-alive client connections.
header
Duration 		(Optional)

Header sets the timeout in the “Keep-Alive: timeout=time” response header field.

ClientSettingsPolicySpec

(Appears on: ClientSettingsPolicy)

ClientSettingsPolicySpec defines the desired state of ClientSettingsPolicy.

Field Description
body
ClientBody 		(Optional)

Body defines the client request body settings.
keepAlive
ClientKeepAlive 		(Optional)

KeepAlive defines the keep-alive settings.
targetRef
sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRef identifies an API object to apply the policy to. Object must be in the same namespace as the policy. Support: Gateway, HTTPRoute, GRPCRoute.

ControllerLogLevel (string alias)

(Appears on: Logging)

ControllerLogLevel type defines the logging level for the control plane.

Value Description

"debug"

ControllerLogLevelDebug is the debug level for control plane logging.

"error"

ControllerLogLevelError is the error level for control plane logging.

"info"

ControllerLogLevelInfo is the info level for control plane logging.

ControllerStatus

(Appears on: AuthenticationFilterStatus, ExternalLoadBalancerStatus, SnippetsFilterStatus)

Field Description
controllerName
sigs.k8s.io/gateway-api/apis/v1.GatewayController

ControllerName is a domain/path string that indicates the name of the controller that wrote this status. This corresponds with the controllerName field on GatewayClass.

Example: “example.net/gateway-controller”.

The format of this field is DOMAIN “/” PATH, where DOMAIN and PATH are valid Kubernetes names (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names).

Controllers MUST populate this field when writing status. Controllers should ensure that entries to status populated with their ControllerName are cleaned up when they are no longer necessary.
conditions
[]Kubernetes meta/v1.Condition 		(Optional)

Conditions describe the status of the resource with respect to this controller.

DefaultLogProfile (string alias)

(Appears on: LogSource)

DefaultLogProfile identifies a built-in WAF log profile bundle.

Value Description

"log_all"

DefaultLogProfileAll logs all events.

"log_blocked"

DefaultLogProfileBlocked logs blocked events.

"log_default"

DefaultLogProfileDefault logs illegal events (equivalent to log_illegal).

"log_grpc_all"

DefaultLogProfileGRPCAll logs all gRPC events.

"log_grpc_blocked"

DefaultLogProfileGRPCBlocked logs blocked gRPC events.

"log_grpc_illegal"

DefaultLogProfileGRPCIllegal logs illegal gRPC events.

"log_illegal"

DefaultLogProfileIllegal logs illegal events.

Duration (string alias)

(Appears on: ClientBody, ClientKeepAlive, ClientKeepAliveTimeout, JWTAuth, OIDCSessionConfig, ProxyTimeout, UpstreamKeepAlive, DNSResolver, TelemetryExporter)

Duration is a string value representing a duration in time. Duration can be specified in milliseconds (ms), seconds (s), minutes (m), hours (h). A value without a suffix is seconds. Examples: 120s, 50ms, 5m, 1h.

ExtProcessConfig

(Appears on: PayloadProcessorEntry)

ExtProcessConfig defines the configuration for an ExtProcess processor that delegates to an external service.

Field Description
authTokenRef
LocalObjectReference

AuthTokenRef is a reference to a Secret containing an authentication token for the external service.
backendRef
sigs.k8s.io/gateway-api/apis/v1.BackendObjectReference

BackendRef is a reference to the external service that will process the payloads. The referenced backend must be a core Service and must specify a port.

ExternalLoadBalancerConditionReason (string alias)

ExternalLoadBalancerConditionReason is a reason for an ExternalLoadBalancer condition type.

Value Description

"Accepted"

ExternalLoadBalancerConditionReasonAccepted is used with the Accepted condition type when the condition is true.

"Conflicted"

ExternalLoadBalancerConditionReasonConflicted is used with the Accepted condition type when another ExternalLoadBalancer already references the same Gateway. A Gateway can be fronted by exactly one external load balancer, so the oldest is accepted and the others are Conflicted.

"Invalid"

ExternalLoadBalancerConditionReasonInvalid is used with the Accepted condition type when the ExternalLoadBalancer is invalid.

ExternalLoadBalancerConditionType (string alias)

ExternalLoadBalancerConditionType is a type of condition associated with ExternalLoadBalancer.

Value Description

"Accepted"

ExternalLoadBalancerConditionTypeAccepted indicates that the ExternalLoadBalancer is accepted.

Possible reasons for this condition to be True:

  • Accepted

Possible reasons for this condition to be False:

  • Invalid
  • Conflicted.

ExternalLoadBalancerSpec

(Appears on: ExternalLoadBalancer)

ExternalLoadBalancerSpec defines the desired state of ExternalLoadBalancer.

Field Description
gatewayLink
GatewayLinkConfig 		(Optional)

GatewayLink configures F5 BIG-IP as the external load balancer using F5 Container Ingress Services. It is the first supported backend. Additional backend types may be added as sibling fields in the future.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies the Gateways this external load balancer applies to. Each object must be in the same namespace as the ExternalLoadBalancer resource. Exactly one Gateway is supported for now. Support: Gateway.

ExternalLoadBalancerStatus

(Appears on: ExternalLoadBalancer)

ExternalLoadBalancerStatus defines the state of ExternalLoadBalancer.

Field Description
controllers
[]ControllerStatus

Controllers is a list of Gateway API controllers that processed the ExternalLoadBalancer and the status of the ExternalLoadBalancer with respect to each controller.

GatewayLinkConfig

(Appears on: ExternalLoadBalancerSpec)

GatewayLinkConfig defines the configuration for integrating with F5 BIG-IP as the external load balancer for NGINX Gateway Fabric using F5 Container Ingress Services. IngressLink API Definition: https://github.com/F5Networks/k8s-bigip-ctlr/blob/master/docs/config_examples/customResourceDefinitions/customresourcedefinitions.yml

Field Description
virtualServerAddress
string 		(Optional)

VirtualServerAddress is the static IP address to configure on BIG-IP for the virtual server. This is mutually exclusive with IPAMLabel.
virtualServerName
string 		(Optional)

VirtualServerName is a custom name for the BIG-IP virtual server.
ipamLabel
string 		(Optional)

IPAMLabel is the label used by F5 IPAM Controller to allocate an IP address. The IPAM controller will assign an IP from the pool associated with this label. This is mutually exclusive with VirtualServerAddress.
host
string 		(Optional)

Host is the hostname for the BIG-IP virtual server.
partition
string 		(Optional)

Partition is the BIG-IP partition where resources will be created. The partition must already exist on BIG-IP and cannot be “Common”.
bigipRouteDomain
int32 		(Optional)

BigIPRouteDomain is the route domain ID for the BIG-IP virtual server.
tls
GatewayLinkTLS 		(Optional)

TLS defines the TLS configuration for the BIG-IP virtual server.
multiCluster
GatewayLinkMultiCluster 		(Optional)

MultiCluster defines the multi-cluster configuration for load balancing traffic across NGINX instances in multiple clusters.
serviceAddress
GatewayLinkServiceAddress 		(Optional)

ServiceAddress configures Layer 3 settings for the BIG-IP virtual server address.
additionalIngressLinkSpec
k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.JSON 		(Optional)

AdditionalIngressLinkSpec is an escape hatch for IngressLink fields that are not yet modeled by GatewayLink. Its contents are merged verbatim into the generated IngressLink spec and are NOT validated by NGINX Gateway Fabric. Fields set here take lower precedence than the explicitly modeled GatewayLink fields above; NGINX Gateway Fabric always sets the IngressLink selector internally and it cannot be overridden through this field. Use with caution since contents bypass schema validation, defaulting, and CEL rules, and flow through to BIG-IP via F5 CIS.
iRules
[]string 		(Optional)

IRules is a list of BIG-IP iRules to apply to the virtual server. Each iRule must be specified using the full path format /partition/irule_name, for example “/Common/Proxy_Protocol_iRule”.
monitors
[]GatewayLinkMonitor 		(Optional)

Monitors is a list of BIG-IP health monitors to associate with the virtual server pool.

GatewayLinkMonitor

(Appears on: GatewayLinkConfig)

GatewayLinkMonitor defines a BIG-IP health monitor reference.

Field Description
name
string

Name is the full path of the health monitor on BIG-IP (e.g., “/Common/http”).
reference
string

Reference specifies the source of the monitor. Currently only “bigip” is supported.

GatewayLinkMultiCluster

(Appears on: GatewayLinkConfig)

GatewayLinkMultiCluster defines the multi-cluster configuration for GatewayLink. When configured, CIS load balances traffic across NGINX instances in multiple clusters. This is set only on the cluster that runs CIS. The other clusters run NGINX with a matching Gateway and Service but not CIS, so they do not set multiCluster. CIS reaches those clusters over a kubeconfig.

Field Description
localClusterName
string

LocalClusterName is the name of this cluster as configured in the CIS deployment via the –local-cluster-name flag. NGINX Gateway Fabric uses it as the cluster name for the local entry in the IngressLink’s multiClusterServices, which points at this cluster’s own Gateway Service. It must match the name CIS knows this cluster by, otherwise CIS cannot resolve the local service.
remoteClusters
[]GatewayLinkRemoteCluster

RemoteClusters is the list of remote clusters that also run NGINX Gateway Fabric.

GatewayLinkRemoteCluster

(Appears on: GatewayLinkMultiCluster)

GatewayLinkRemoteCluster defines a remote cluster for multi-cluster load balancing.

Field Description
clusterName
string

ClusterName is one of the names of the remote clusters as configured in the CIS deployment.
namespace
string 		(Optional)

Namespace is the namespace of the NGINX service in the remote cluster. If not specified, defaults to the local Gateway’s namespace.
service
string 		(Optional)

Service is the name of the NGINX service in the remote cluster. If not specified, defaults to the local Gateway’s service name.
weight
int32 		(Optional)

Weight is the load balancing weight for this cluster’s service.

GatewayLinkServiceAddress

(Appears on: GatewayLinkConfig)

GatewayLinkServiceAddress configures Layer 3 settings for the BIG-IP virtual server address.

Field Description
icmpEcho
ICMPEcho 		(Optional)

ICMPEcho controls whether the virtual server address responds to ICMP echo (ping).
trafficGroup
string 		(Optional)

TrafficGroup is the BIG-IP traffic group that owns the virtual server address, in the full path format, for example “/Common/traffic-group-test”.

GatewayLinkTLS

(Appears on: GatewayLinkConfig)

GatewayLinkTLS defines the TLS configuration for the BIG-IP virtual server.

Field Description
reference
TLSReferenceType 		(Optional)

Reference specifies the source of the SSL profiles. “bigip” means the profiles already exist on BIG-IP. “secret” means they come from Kubernetes secrets of type kubernetes.io/tls. If not specified, defaults to “bigip”.
clientSSLs
[]string 		(Optional)

ClientSSLs is a list of client SSL profiles that BIG-IP uses to terminate TLS from the client. When reference is “bigip”, each entry is the full path of a profile on BIG-IP in the form /partition/profile_name, for example /Common/clientssl. When reference is “secret”, each entry is the name of a Kubernetes secret of type kubernetes.io/tls that holds the certificate and key.
serverSSLs
[]string 		(Optional)

ServerSSLs is a list of server SSL profiles that BIG-IP uses to re-encrypt traffic to NGINX. When reference is “bigip”, each entry is the full path of a profile on BIG-IP in the form /partition/profile_name, for example /Common/serverssl. When reference is “secret”, each entry is the name of a Kubernetes secret of type kubernetes.io/tls that holds the certificate and key.

HTTPBundleSource

(Appears on: LogSource, PolicySource)

HTTPBundleSource configures direct bundle fetching from an HTTP/HTTPS URL.

Field Description
url
string

URL is the full URL of the compiled policy bundle (.tgz), e.g. https://storage.example.com/bundles/policy.tgz.

HashMethodKey (string alias)

(Appears on: UpstreamSettingsPolicySpec)

HashMethodKey defines the key used for hash-based load balancing methods. The key must be a valid NGINX variable name starting with ‘$’ followed by lowercase letters and underscores only. For a full list of NGINX variables, refer to: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables

ICMPEcho (string alias)

(Appears on: GatewayLinkServiceAddress)

ICMPEcho controls whether the BIG-IP virtual server address responds to ICMP echo.

Value Description

"disable"

ICMPEchoDisable means the virtual server address never responds to ICMP echo.

"enable"

ICMPEchoEnable means the virtual server address always responds to ICMP echo.

"selective"

ICMPEchoSelective means BIG-IP responds to ICMP echo based on the state of the virtual server.

JWTAuth

(Appears on: AuthenticationFilterSpec)

JWTAuth configures JWT-based authentication (NGINX Plus).

Field Description
file
JWTFileKeySource 		(Optional)

File specifies local JWKS configuration. Required when Source == File.
keyCache
Duration 		(Optional)

KeyCache is the cache duration for keys. Configures auth_jwt_key_cache directive. https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_cache Example: “auth_jwt_key_cache 10m;”.
remote
JWTRemoteKeySource 		(Optional)

Remote specifies remote JWKS configuration. Required when Source == Remote.
authorization
Authorization 		(Optional)

Authorization defines the authorization (authz) specification. Enables configuration of token claim validation.
leeway
Duration 		(Optional)

Leeway is the acceptable clock skew for exp & nbf claims. If exp & nbf claims are not defined, this directive takes no effect. Configures auth_jwt_leeway directive. https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_leeway Example: “auth_jwt_leeway 60s”. Default: 0s.
realm
string

Realm used by NGINX auth_jwt directive https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt Configures “realm=”” in WWW-Authenticate header in error page location.
source
JWTKeySource

Source selects how JWT keys are provided: local file or remote JWKS.

JWTFileKeySource

(Appears on: JWTAuth)

JWTFileKeySource specifies local JWKS key configuration.

Field Description
secretRef
LocalObjectReference

SecretRef references a Secret containing the JWKS.

JWTKeySource (string alias)

(Appears on: JWTAuth)

JWTKeySource specifies the source of the keys used to verify JWT signatures.

Value Description

"File"

JWTKeySourceFile configures JWT to fetch JWKS from a local secret.

"Remote"

JWTKeySourceRemote configures JWT to fetch JWKS from a remote source.

JWTRemoteKeySource

(Appears on: JWTAuth)

JWTRemoteKeySource specifies remote JWKS configuration.

Field Description
uri
string

URI is the JWKS endpoint.
caCertificateRefs
[]LocalObjectReference 		(Optional)

CACertificateRefs references a list of secrets containing trusted CA certificates in PEM format used to verify the server certificate of the JWKS endpoint. The referenced secrets must contain an entry with the key “ca.crt”. Only one secret can be referenced currently. If not specified, the system CA bundle is used.

Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_trusted_certificate

LoadBalancingType (string alias)

(Appears on: UpstreamSettingsPolicySpec)

LoadBalancingType defines the supported load balancing methods.

Value Description

"hash"

LoadBalancingTypeHash enables generic hash-based load balancing, routing requests to upstream servers based on a hash of a specified key HashMethodKey field must be set when this method is selected. Example configuration: hash $binary_remote_addr;.

"hash consistent"

LoadBalancingTypeHashConsistent enables consistent hash-based load balancing, which minimizes the number of keys remapped when a server is added or removed. HashMethodKey field must be set when this method is selected. Example configuration: hash $binary_remote_addr consistent;.

"ip_hash"

LoadBalancingTypeIPHash enables IP hash-based load balancing, ensuring requests from the same client IP are routed to the same upstream server.

"least_conn"

LoadBalancingTypeLeastConnection enables least-connections load balancing, routing requests to the upstream server with the fewest active connections.

"least_time header"

LoadBalancingTypeLeastTimeHeader enables least-time load balancing, routing requests to the upstream server with the least time to receive the response header.

"least_time header inflight"

LoadBalancingTypeLeastTimeHeaderInflight enables least-time load balancing, routing requests to the upstream server with the least time to receive the response header, considering the incomplete requests.

"least_time last_byte"

LoadBalancingTypeLeastTimeLastByte enables least-time load balancing, routing requests to the upstream server with the least time to receive the full response.

"least_time last_byte inflight"

LoadBalancingTypeLeastTimeLastByteInflight enables least-time load balancing, routing requests to the upstream server with the least time to receive the full response, considering the incomplete requests.

"random"

LoadBalancingTypeRandom enables random load balancing, routing requests to upstream servers in a random manner.

"random two"

LoadBalancingTypeRandomTwo enables a variation of random load balancing that randomly selects two servers and forwards traffic to one of them. The default method is least_conn which passes a request to a server with the least number of active connections.

"random two least_conn"

LoadBalancingTypeRandomTwoLeastConnection enables a variation of least-connections balancing that randomly selects two servers and forwards traffic to the one with fewer active connections.

"random two least_time=header"

LoadBalancingTypeRandomTwoLeastTimeHeader enables a variation of least-time load balancing that randomly selects two servers and forwards traffic to the one with the least time to receive the response header.

"random two least_time=last_byte"

LoadBalancingTypeRandomTwoLeastTimeLastByte enables a variation of least-time load balancing that randomly selects two servers and forwards traffic to the one with the least time to receive the full response.

"round_robin"

LoadBalancingTypeRoundRobin enables round-robin load balancing, distributing requests evenly across all upstream servers.

LocalObjectReference

(Appears on: BasicAuth, BundleAuth, ExtProcessConfig, JWTFileKeySource, JWTRemoteKeySource, LogSource, OIDCAuth, PolicySource)

LocalObjectReference specifies a local Kubernetes object.

Field Description
name
string

Name is the name of the referenced object.

LocalRateLimit

(Appears on: RateLimit)

LocalRateLimit contains the local rate limit rules.

Field Description
rules
[]RateLimitRule 		(Optional)

Rules contains the list of rate limit rules.

LogRef

(Appears on: WAFSecurityLog)

LogRef holds all CRD-backed log profile references.

Field Description
apLogConfRef
APLogConfReference 		(Optional)

APLogConfRef references an APLogConf CRD compiled by PLM. Cross-namespace references require a ReferenceGrant.

LogSource

(Appears on: WAFSecurityLog)

LogSource holds all non-CRD configuration for fetching a WAF log profile bundle. Exactly one of DefaultProfile, HTTPSource, NIMSource, or N1CSource must be set.

Field Description
defaultProfile
DefaultLogProfile 		(Optional)

DefaultProfile selects one of the built-in WAF log profile bundles shipped with the WAF engine. Mutually exclusive with HTTPSource, NIMSource, and N1CSource.
httpSource
HTTPBundleSource 		(Optional)

HTTPSource configures direct bundle fetching from an HTTP/HTTPS URL. Mutually exclusive with DefaultProfile, NIMSource and N1CSource.
nimSource
NIMLogProfileBundleSource 		(Optional)

NIMSource configures bundle fetching from NGINX Instance Manager. Mutually exclusive with DefaultProfile, HTTPSource and N1CSource.
n1cSource
N1CLogProfileBundleSource 		(Optional)

N1CSource configures bundle fetching from F5 NGINX One Console. Mutually exclusive with DefaultProfile, HTTPSource, and NIMSource.
auth
BundleAuth 		(Optional)

Auth configures authentication credentials for fetching the log bundle. Only applicable when url is set.
tlsSecret
LocalObjectReference 		(Optional)

TLSSecretRef references a Secret containing a custom CA certificate (key: “ca.crt”). Only applicable when url is set.
validation
BundleValidation 		(Optional)

Validation configures integrity verification for the downloaded log bundle. Only applicable when url is set.
polling
BundlePolling 		(Optional)

Polling configures automatic periodic re-fetching of the log bundle. Only applicable when url is set.
timeout
Kubernetes meta/v1.Duration 		(Optional)

Timeout is the maximum duration for a single log bundle fetch attempt. Defaults to 30s when not set. Only applicable when url is set.
retryAttempts
int32

RetryAttempts is the maximum number of additional fetch attempts on transient failures (network errors, HTTP 5xx). Set to 0 to disable retries. Defaults to 3. Non-transient errors (HTTP 4xx, checksum mismatch) are never retried. Only applicable when url is set.
insecureSkipVerify
bool 		(Optional)

InsecureSkipVerify disables TLS certificate verification when fetching the bundle. Not recommended for production use.

Logging

(Appears on: NginxGatewaySpec)

Logging defines logging related settings for the control plane.

Field Description
level
ControllerLogLevel 		(Optional)

Level defines the logging level.

N1CBundleSource

(Appears on: PolicySource)

N1CBundleSource configures bundle fetching from F5 NGINX One Console (N1C). Exactly one of policyName or policyObjectID must be set.

Field Description
policyName
string 		(Optional)

PolicyName is the name of the security policy in N1C. Mutually exclusive with policyObjectID.
policyObjectID
string 		(Optional)

PolicyObjectID is the unique object identifier of the security policy in N1C (e.g. “pol_-IUuEUN7ST63oRC7AlQPLw”). Mutually exclusive with policyName.
policyVersionID
string 		(Optional)

PolicyVersionID pins a specific version of the policy bundle using its opaque version ID (e.g. “pv_UJ2gL5fOQ3Gnb3OVuVo1XA”). When omitted, the latest available version is used.
url
string

URL is the base URL of the F5 NGINX One Console instance, e.g. https://.volterra.us.
namespace
string

Namespace is the NGINX One Console namespace that owns the security policy.

N1CLogProfileBundleSource

(Appears on: LogSource)

N1CLogProfileBundleSource configures log profile bundle fetching from F5 NGINX One Console (N1C). Exactly one of profileName or profileObjectID must be set.

Field Description
profileName
string

ProfileName is the name of the log profile in N1C that corresponds to the log profile bundle.
profileObjectID
string

ProfileObjectID is the unique object identifier of the log profile in N1C (e.g. “lp_8s8uZxLpThWwEGF7LTn_rA”) that corresponds to the log profile bundle.
url
string

URL is the base URL of the F5 NGINX One Console instance, e.g. https://.volterra.us.
namespace
string

Namespace is the NGINX One Console namespace that owns the log profile.

NIMBundleSource

(Appears on: PolicySource)

NIMBundleSource configures bundle fetching from NGINX Instance Manager (NIM). Exactly one of policyName or policyUID must be set.

Field Description
policyName
string 		(Optional)

PolicyName is the name of the compiled policy bundle in NIM. Mutually exclusive with policyUID.
policyUID
string 		(Optional)

PolicyUID is the unique identifier of the compiled policy bundle in NIM. Mutually exclusive with policyName. Must be a valid UUID (e.g. “2bc1e3ac-7990-4ca4-910a-8634c444c804”).
url
string

URL is the base URL of the NGINX Instance Manager instance, e.g. https://nim.example.com.

NIMLogProfileBundleSource

(Appears on: LogSource)

NIMLogProfileBundleSource configures log profile bundle fetching from NGINX Instance Manager (NIM).

Field Description
profileName
string

ProfileName is the name of the compiled log profile bundle in NIM.
url
string

URL is the base URL of the NGINX Instance Manager instance, e.g. https://nim.example.com.

NginxContext (string alias)

(Appears on: Snippet)

NginxContext represents the NGINX configuration context.

Value Description

"http"

NginxContextHTTP is the http context of the NGINX configuration. https://nginx.org/en/docs/http/ngx_http_core_module.html#http

"http.server"

NginxContextHTTPServer is the server context of the NGINX configuration. https://nginx.org/en/docs/http/ngx_http_core_module.html#server

"http.server.location"

NginxContextHTTPServerLocation is the location context of the NGINX configuration. https://nginx.org/en/docs/http/ngx_http_core_module.html#location

"main"

NginxContextMain is the main context of the NGINX configuration.

NginxGatewayConditionReason (string alias)

NginxGatewayConditionReason defines the set of reasons that explain why a particular NginxGateway condition type has been raised.

Value Description

"Invalid"

NginxGatewayReasonInvalid is a reason that is used with the “Valid” condition when the condition is False.

"Valid"

NginxGatewayReasonValid is a reason that is used with the “Valid” condition when the condition is True.

NginxGatewayConditionType (string alias)

NginxGatewayConditionType is a type of condition associated with an NginxGateway. This type should be used with the NginxGatewayStatus.Conditions field.

Value Description

"Valid"

NginxGatewayConditionValid is a condition that is true when the NginxGateway configuration is syntactically and semantically valid.

NginxGatewaySpec

(Appears on: NginxGateway)

NginxGatewaySpec defines the desired state of the NginxGateway.

Field Description
logging
Logging 		(Optional)

Logging defines logging related settings for the control plane.

NginxGatewayStatus

(Appears on: NginxGateway)

NginxGatewayStatus defines the state of the NginxGateway.

Field Description
conditions
[]Kubernetes meta/v1.Condition 		(Optional)

OIDCAuth

(Appears on: AuthenticationFilterSpec)

OIDCAuth configures OpenID Connect Authentication. Only available for NGINX Plus users.

Field Description
extraAuthArgs
map[string]string 		(Optional)

ExtraAuthArgs sets additional query arguments for the authentication request URL. Arguments are appended with “&”. For example: “prompt=consent&audience=api”. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#extra_auth_args
crlSecretRef
LocalObjectReference 		(Optional)

CRLSecretRef references a Secret containing a certificate revocation list in PEM format. The referenced Secret must contain an entry with the key “ca.crl”. This is used to verify that certificates presented by the OpenID Provider endpoints have not been revoked.
configURL
string 		(Optional)

ConfigURL sets a custom URL to retrieve the OpenID Provider metadata. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#config_url NGINX Default: /.well-known/openid-configuration
pkce
bool 		(Optional)

PKCE enables Proof Key for Code Exchange (PKCE) for the authentication flow. If nil, NGINX automatically enables PKCE when the OpenID Provider requires it. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#pkce
session
OIDCSessionConfig 		(Optional)

Session configures session management for OIDC authentication.
logout
OIDCLogoutConfig 		(Optional)

Logout defines the logout behavior for OIDC authentication.
redirectURI
string 		(Optional)

RedirectURI sets a custom redirect URI for the OIDC callback. If a path-only URI is specified, a callback location block is created to handle the redirect from the OIDC provider. If a full URI is specified, it points to an external callback handler; no location block is created. If not specified, defaults to /oidccallback_. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#redirect_uri NGINX Default: /oidc_callback Example: /oidc_callback, https://cafe.example.com:8442/oidc_callback
authorization
Authorization 		(Optional)

Authorization defines the authorization (authz) specification. Enables configuration of token claim validation.
clientSecretRef
LocalObjectReference

ClientSecretRef references a Kubernetes secret which contains the OIDC client secret to be used in the Authentication Request: https://openid.net/specs/openid-connect-core-1_0.html#AuthRequest. The referenced Secret must contain an entry with the key “client-secret”. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_secret
issuer
string

Issuer is the URL of the OpenID Provider. Must exactly match the “issuer” value from the provider’s .well-known/openid-configuration endpoint. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#issuer Examples: - Keycloak: https://keycloak.example.com/realms/my-realm - Okta: https://dev-123456.okta.com/oauth2/default - Auth0: https://my-tenant.auth0.com/
clientID
string

ClientID is the client identifier registered with the OpenID Provider. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#client_id
caCertificateRefs
[]LocalObjectReference 		(Optional)

CACertificateRefs references a list of secrets containing trusted CA certificates in PEM format used to verify the certificates of the OpenID Provider endpoints. The referenced secrets must contain an entry with the key “ca.crt”. Only one secret can be referenced currently. If not specified, the system CA bundle is used.

Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#ssl_trusted_certificate NGINX Default: system CA bundle

OIDCLogoutConfig

(Appears on: OIDCAuth)

OIDCLogoutConfig defines the logout behavior for OIDC authentication.

Field Description
uri
string 		(Optional)

URI defines the path for initiating session logout. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#logout_uri Example: /logout
postLogoutURI
string 		(Optional)

PostLogoutURI defines the URI to redirect to after logout. Must match the configuration on the provider’s side. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#post_logout_uri Example: /after_logout, https://example.com/after_logout
frontChannelLogoutURI
string 		(Optional)

FrontChannelLogoutURI defines the path for front-channel logout. The OpenID Provider should be configured to set “iss” and “sid” arguments. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#frontchannel_logout_uri Example: /frontchannel_logout
tokenHint
bool 		(Optional)

TokenHint adds the id_token_hint argument to the provider’s Logout Endpoint. Some OpenID Providers require this. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#logout_token_hint NGINX Default: false

OIDCSessionConfig

(Appears on: OIDCAuth)

OIDCSessionConfig configures session management for OIDC authentication.

Field Description
cookieName
string 		(Optional)

CookieName sets the name of the session cookie. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#cookie_name NGINX Default: NGX_OIDC_SESSION
timeout
Duration 		(Optional)

Timeout sets the session timeout duration. Directive: https://nginx.org/en/docs/http/ngx_http_oidc_module.html#session_timeout NGINX Default: 8h

PayloadProcessorEntry

(Appears on: PayloadProcessorSpec)

PayloadProcessorEntry defines a single processing step in the pipeline.

Field Description
extProcess
ExtProcessConfig 		(Optional)

ExtProcess defines the configuration for an ExtProcess processor that delegates to an external service.
type
ProcessorType

Type specifies how the processor executes. ExtProcess calls an external service.

PayloadProcessorSpec

(Appears on: PayloadProcessor)

PayloadProcessorSpec defines the desired state of a PayloadProcessor.

Field Description
targetRef
sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRef identifies the Gateway or HTTPRoute this policy applies to. Objects must be in the same namespace as the policy. Follows the standard policy attachment pattern (GEP-713).

Support: Gateway, HTTPRoute
processors
[]PayloadProcessorEntry

Processors is an ordered list of processing steps to be applied to the request and response payloads. It is currently limited to a single processor (MaxItems=1); the list form is reserved for future multi-processor pipelines.

PolicyRef

(Appears on: WAFPolicySpec)

PolicyRef holds all CRD-backed policy references.

Field Description
apPolicyRef
APPolicyReference 		(Optional)

APPolicyRef references an APPolicy CRD compiled by PLM. Cross-namespace references require a ReferenceGrant.

PolicySource

(Appears on: WAFPolicySpec)

PolicySource holds all non-CRD configuration for fetching a WAF policy bundle.

Field Description
httpSource
HTTPBundleSource 		(Optional)

HTTPSource configures direct bundle fetching from an HTTP/HTTPS URL. Required when type is HTTP; must not be set for other types.
nimSource
NIMBundleSource 		(Optional)

NIMSource configures bundle fetching from NGINX Instance Manager. Required when type is NIM; must not be set for other types.
n1cSource
N1CBundleSource 		(Optional)

N1CSource configures bundle fetching from F5 NGINX One Console. Required when type is N1C; must not be set for other types.
auth
BundleAuth 		(Optional)

Auth configures authentication credentials for fetching the bundle.
tlsSecret
LocalObjectReference 		(Optional)

TLSSecretRef references a Secret containing a custom CA certificate (key: “ca.crt”) for verifying the bundle server’s TLS certificate.
validation
BundleValidation 		(Optional)

Validation configures integrity verification for the downloaded bundle.
polling
BundlePolling 		(Optional)

Polling configures automatic periodic re-fetching of the bundle.
timeout
Kubernetes meta/v1.Duration 		(Optional)

Timeout is the maximum duration for a single bundle fetch attempt. Defaults to 30s when not set.
retryAttempts
int32

RetryAttempts is the maximum number of additional fetch attempts on transient failures (network errors, HTTP 5xx). Set to 0 to disable retries. Defaults to 3. Non-transient errors (HTTP 4xx, checksum mismatch) are never retried.
insecureSkipVerify
bool 		(Optional)

InsecureSkipVerify disables TLS certificate verification when fetching the bundle. Not recommended for production use.

PolicySourceType (string alias)

(Appears on: WAFPolicySpec)

PolicySourceType identifies the source type for a WAF bundle.

Value Description

"HTTP"

PolicySourceTypeHTTP fetches a compiled .tgz bundle directly from an HTTP/HTTPS URL.

"N1C"

PolicySourceTypeN1C fetches a compiled bundle from the F5 NGINX One Console security policies API. Requires managedSource.n1cNamespace in addition to managedSource.policyName. Authentication uses the APIToken scheme: the “token” key from the referenced Secret is sent as “Authorization: APIToken ”.

"NIM"

PolicySourceTypeNIM fetches a compiled bundle from the NGINX Instance Manager security policies API.

"PLM"

PolicySourceTypePLM references an APPolicy CRD managed by the Policy Lifecycle Manager (PLM). Bundles are fetched from PLM’s S3-compatible storage (SeaweedFS). Cluster-wide S3 connection parameters are configured via CLI flags (–plm-storage-*).

ProcessorType (string alias)

(Appears on: PayloadProcessorEntry)

ProcessorType specifies how the processor executes. ExtProcess calls an external service.

Value Description

"ExtProcess"

ProcessorTypeExtProcess delegates processing to an external service.

ProxyBuffering

(Appears on: ProxySettingsPolicySpec)

ProxyBuffering contains the settings for proxy buffering.

Field Description
disable
bool 		(Optional)

Disable enables or disables buffering of responses from the proxied server. If Disable is true, buffering is disabled. If Disable is false, or if Disable is not set, buffering is enabled. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
bufferSize
Size 		(Optional)

BufferSize sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size
buffers
ProxyBuffers 		(Optional)

Buffers sets the number and size of buffers used for reading a response from the proxied server, for a single connection. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers
busyBuffersSize
Size 		(Optional)

BusyBuffersSize sets the total size of buffers that can be busy sending a response to the client, while the response is not yet fully read. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size

ProxyBuffers

(Appears on: ProxyBuffering)

ProxyBuffers defines the number and size of the proxy buffers.

Field Description
size
Size

Size sets the size of each buffer.
number
int32

Number sets the number of buffers.

ProxySettingsPolicySpec

(Appears on: ProxySettingsPolicy)

ProxySettingsPolicySpec defines the desired state of the ProxySettingsPolicy.

Field Description
buffering
ProxyBuffering 		(Optional)

Buffering configures the buffering of responses from the proxied server.
timeout
ProxyTimeout 		(Optional)

Timeout configures timeouts for the connection to the proxied server.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies the API object(s) to apply the policy to. Objects must be in the same namespace as the policy. Support: Gateway, HTTPRoute, GRPCRoute

ProxyTimeout

(Appears on: ProxySettingsPolicySpec)

ProxyTimeout defines timeout settings for the connection to the proxied server.

Field Description
connect
Duration 		(Optional)

Connect sets the timeout for establishing a connection with the proxied server. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout
read
Duration 		(Optional)

Read sets the timeout for reading a response from the proxied server. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
send
Duration 		(Optional)

Send sets the timeout for transmitting a request to the proxied server. Directive: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout

Rate (string alias)

(Appears on: RateLimitRule)

Rate is a string value representing a rate. Rate can be specified in r/s or r/m.

RateLimit

(Appears on: RateLimitPolicySpec)

RateLimit contains settings for Rate Limiting.

Field Description
local
LocalRateLimit 		(Optional)

Local defines the local rate limit rules for this policy.
dryRun
bool 		(Optional)

DryRun enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_dry_run
logLevel
RateLimitLogLevel 		(Optional)

LogLevel sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are info, notice, warn or error.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_log_level
rejectCode
int32 		(Optional)

RejectCode sets the status code to return in response to rejected requests. Must fall into the range 400-599.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_status

RateLimitLogLevel (string alias)

(Appears on: RateLimit)

RateLimitLogLevel defines the log level for cases when the server refuses to process requests due to rate exceeding, or delays request processing.

Value Description

"error"

RateLimitLogLevelError is the error level rate limit logs.

"info"

RateLimitLogLevelInfo is the info level rate limit logs.

"notice"

RateLimitLogLevelNotice is the notice level rate limit logs.

"warn"

RateLimitLogLevelWarn is the warn level rate limit logs.

RateLimitPolicySpec

(Appears on: RateLimitPolicy)

RateLimitPolicySpec defines the desired state of the RateLimitPolicy.

Field Description
rateLimit
RateLimit 		(Optional)

RateLimit defines the Rate Limit settings.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to. Objects must be in the same namespace as the policy.

Support: Gateway, HTTPRoute, GRPCRoute

RateLimitRule

(Appears on: LocalRateLimit)

RateLimitRule contains settings for a RateLimit Rule.

Field Description
zoneSize
Size 		(Optional)

ZoneSize is the size of the shared memory zone.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone
delay
int32 		(Optional)

Delay specifies a limit at which excessive requests become delayed. Default value is zero, which means all excessive requests are delayed.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req
noDelay
bool 		(Optional)

NoDelay disables the delaying of excessive requests while requests are being limited. NoDelay cannot be true when Delay is also set.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req
burst
int32 		(Optional)

Burst sets the maximum burst size of requests. If the requests rate exceeds the rate configured for a zone, their processing is delayed such that requests are processed at a defined rate. Excessive requests are delayed until their number exceeds the maximum burst size in which case the request is terminated with an error.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req
rate
Rate

Rate represents the rate of requests permitted. The rate is specified in requests per second (r/s) or requests per minute (r/m).

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone
key
string

Key represents the key to which the rate limit is applied. The key can contain text, variables, and their combination.

Directive: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone

RequireType (string alias)

(Appears on: Authorization, Rule)

RequireType defines how JWT Claims are validated.

Value Description

"All"

RequireTypeAll authorizes claims that satisfy all requirements.

"Any"

RequireTypeAny authorizes claims that satisfy any requirement.

Rule

(Appears on: Authorization)

Rule defines a list of claims, and authorization rules for those claims.

Field Description
require
RequireType 		(Optional)

Require sets the authorization mode for a specific claim within a rule. When set to All, a token’s claim must match all values within that claim. When set to Any, a token’s claim must match at least one value with that claim.
claims
[]Claim

Claims defines a list of claims required by users.

SecurityLogDestination

(Appears on: WAFSecurityLog)

SecurityLogDestination defines the destination for security logs.

Field Description
file
SecurityLogFile 		(Optional)

File defines the file destination configuration. Only valid when type is “file”.
syslog
SecurityLogSyslog 		(Optional)

Syslog defines the syslog destination configuration. Only valid when type is “syslog”.
type
SecurityLogDestinationType

Type identifies the type of security log destination.

SecurityLogDestinationType (string alias)

(Appears on: SecurityLogDestination)

SecurityLogDestinationType defines the supported security log destination types.

Value Description

"file"

SecurityLogDestinationTypeFile writes logs to a specified file path.

"stderr"

SecurityLogDestinationTypeStderr outputs logs to container stderr.

"syslog"

SecurityLogDestinationTypeSyslog sends logs to a syslog server via TCP.

SecurityLogFile

(Appears on: SecurityLogDestination)

SecurityLogFile defines the file destination configuration for security logs.

Field Description
path
string

Path is the file path where security logs will be written. Must be accessible to the waf-enforcer container.

SecurityLogSyslog

(Appears on: SecurityLogDestination)

SecurityLogSyslog defines the syslog destination configuration for security logs.

Field Description
server
string

Server is the syslog server address in the format “host:port”.

Size (string alias)

(Appears on: ClientBody, ProxyBuffering, ProxyBuffers, RateLimitRule, UpstreamSettingsPolicySpec, CompressionBuffers, NginxProxySpec)

Size is a string value representing a size. Size can be specified in bytes, kilobytes (k), megabytes (m), or gigabytes (g). Examples: 1024, 8k, 1m.

Snippet

(Appears on: SnippetsFilterSpec, SnippetsPolicySpec)

Snippet represents an NGINX configuration snippet.

Field Description
context
NginxContext

Context is the NGINX context to insert the snippet into.
value
string

Value is the NGINX configuration snippet.

SnippetsFilterConditionReason (string alias)

SnippetsFilterConditionReason is a reason for a SnippetsFilter condition type.

Value Description

"Accepted"

SnippetsFilterConditionReasonAccepted is used with the Accepted condition type when the condition is true.

"Invalid"

SnippetsFilterConditionReasonInvalid is used with the Accepted condition type when SnippetsFilter is invalid.

SnippetsFilterConditionType (string alias)

SnippetsFilterConditionType is a type of condition associated with SnippetsFilter.

Value Description

"Accepted"

SnippetsFilterConditionTypeAccepted indicates that the SnippetsFilter is accepted.

Possible reasons for this condition to be True:

  • Accepted

Possible reasons for this condition to be False:

  • Invalid.

SnippetsFilterSpec

(Appears on: SnippetsFilter)

SnippetsFilterSpec defines the desired state of the SnippetsFilter.

Field Description
snippets
[]Snippet

Snippets is a list of NGINX configuration snippets. There can only be one snippet per context. Allowed contexts: main, http, http.server, http.server.location.

SnippetsFilterStatus

(Appears on: SnippetsFilter)

SnippetsFilterStatus defines the state of SnippetsFilter.

Field Description
controllers
[]ControllerStatus

Controllers is a list of Gateway API controllers that processed the SnippetsFilter and the status of the SnippetsFilter with respect to each controller.

SnippetsPolicySpec

(Appears on: SnippetsPolicy)

SnippetsPolicySpec defines the desired state of the SnippetsPolicy.

Field Description
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to.
snippets
[]Snippet 		(Optional)

Snippets is a list of snippets to be injected into the NGINX configuration.

SpanAttribute

(Appears on: Telemetry, Tracing)

SpanAttribute is a key value pair to be added to a tracing span.

Field Description
key
string

Key is the key for a span attribute. Format: must have all ‘“’ escaped and must not contain any ‘$’ or end with an unescaped ‘\’
value
string

Value is the value for a span attribute. Format: must have all ‘“’ escaped and must not contain any ‘$’ or end with an unescaped ‘\’

TLSReferenceType (string alias)

(Appears on: GatewayLinkTLS)

TLSReferenceType specifies where the BIG-IP SSL profiles come from.

Value Description

"bigip"

TLSReferenceBigIP means the SSL profiles already exist on BIG-IP.

"secret"

TLSReferenceSecret means the SSL profiles are sourced from Kubernetes secrets.

UpstreamKeepAlive

(Appears on: UpstreamSettingsPolicySpec)

UpstreamKeepAlive defines the keep-alive settings for upstreams.

Field Description
connections
int32 		(Optional)

Connections sets the maximum number of idle keep-alive connections to upstream servers that are preserved in the cache of each nginx worker process. When this number is exceeded, the least recently used connections are closed. The keepAlive directive for upstreams defaults to 32. To override this value, set the connections field. To disable the keepAlive directive, set connections to 0. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
requests
int32 		(Optional)

Requests sets the maximum number of requests that can be served through one keep-alive connection. After the maximum number of requests are made, the connection is closed. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests
time
Duration 		(Optional)

Time defines the maximum time during which requests can be processed through one keep-alive connection. After this time is reached, the connection is closed following the subsequent request processing. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time
timeout
Duration 		(Optional)

Timeout defines the keep-alive timeout for upstreams. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout

UpstreamSettingsPolicySpec

(Appears on: UpstreamSettingsPolicy)

UpstreamSettingsPolicySpec defines the desired state of the UpstreamSettingsPolicy.

Field Description
zoneSize
Size 		(Optional)

ZoneSize is the size of the shared memory zone used by the upstream. This memory zone is used to share the upstream configuration between nginx worker processes. The more servers that an upstream has, the larger memory zone is required. Default: OSS: 512k, Plus: 1m. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone
keepAlive
UpstreamKeepAlive 		(Optional)

KeepAlive defines the keep-alive settings.
loadBalancingMethod
LoadBalancingType 		(Optional)

LoadBalancingMethod specifies the load balancing algorithm to be used for the upstream. If not specified, NGINX Gateway Fabric defaults to random two least_conn, which differs from the standard NGINX default round-robin.
hashMethodKey
HashMethodKey 		(Optional)

HashMethodKey defines the key used for hash-based load balancing methods. This field is required when LoadBalancingMethod is set to hash or hash consistent.
useClusterIP
bool 		(Optional)

UseClusterIP configures NGINX to route to the Service ClusterIP and port instead of individual Pod IPs. When enabled, NGINX will target a single upstream server corresponding to the Service’s ClusterIP, which is useful for service mesh compatibility and other Kubernetes controllers/operators that require traffic to traverse the Service VIP. This setting applies only when the target Service has a ClusterIP. For headless Services (ClusterIP: None) and ExternalName Services, normal endpoint resolution is used instead. This setting is also not applied to L4/stream upstreams. Defaults to false.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to. Objects must be in the same namespace as the policy. Support: Service

TargetRefs must be distinct. The name field must be unique for all targetRef entries in the UpstreamSettingsPolicy.

WAFPolicySpec

(Appears on: WAFPolicy)

WAFPolicySpec defines the desired state of a WAFPolicy.

Field Description
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies API object(s) to apply the policy to. Objects must be in the same namespace as the policy. All targets must be of the same Kind (all Gateways OR all HTTPRoutes OR all GRPCRoutes). Support: Gateway, HTTPRoute, GRPCRoute.
type
PolicySourceType

Type identifies the source type for the policy bundle. HTTP fetches directly from a URL; NIM uses the NGINX Instance Manager bundles API; N1C uses the F5 NGINX One Console security policies API; PLM references an APPolicy CRD managed by the Policy Lifecycle Manager.
policySource
PolicySource 		(Optional)

PolicySource holds all non-CRD bundle fetch configuration. Used for HTTP, NIM, and N1C policy types. Must not be set when type is PLM.
policyRef
PolicyRef 		(Optional)

PolicyRef holds all CRD-backed policy references. Used for the PLM policy type.
securityLogs
[]WAFSecurityLog 		(Optional)

SecurityLogs defines security logging configurations.

WAFSecurityLog

(Appears on: WAFPolicySpec)

WAFSecurityLog defines security logging configuration for app_protect_security_log directives. Exactly one of logSource or logRef must be set.

Field Description
logSource
LogSource 		(Optional)

LogSource configures all non-CRD log profile bundle sources for this log entry. Used for defaultProfile, httpSource, nimSource, and n1cSource. Must not be set when logRef is used.
logRef
LogRef 		(Optional)

LogRef configures all CRD-backed log profile references for this log entry. Used for PLM-backed APLogConf references.
destination
SecurityLogDestination

Destination defines where security logs are sent.

gateway.nginx.org/v1alpha2

Package v1alpha2 contains API Schema definitions for the gateway.nginx.org API group.

Resource Types:

NginxProxy

NginxProxy is a configuration object that can be referenced from a GatewayClass parametersRef or a Gateway infrastructure.parametersRef. It provides a way to configure data plane settings. If referenced from a GatewayClass, the settings apply to all Gateways attached to the GatewayClass. If referenced from a Gateway, the settings apply to that Gateway alone. If both a Gateway and its GatewayClass reference an NginxProxy, the settings are merged. Settings specified on the Gateway NginxProxy override those set on the GatewayClass NginxProxy.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha2
kind
string 		NginxProxy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
NginxProxySpec

Spec defines the desired state of the NginxProxy.



ipFamily
IPFamilyType 		(Optional)

IPFamily specifies the IP family to be used by the NGINX. If not set, NGF inspects the default/kubernetes Service’s ipFamilies field at startup to obtain the IP family of the cluster and configure NGINX accordingly.
telemetry
Telemetry 		(Optional)

Telemetry specifies the OpenTelemetry configuration.
metrics
Metrics 		(Optional)

Metrics defines the configuration for Prometheus scraping metrics. Changing this value results in a re-roll of the NGINX deployment.
rewriteClientIP
RewriteClientIP 		(Optional)

RewriteClientIP defines configuration for rewriting the client IP to the original client’s IP.
logging
NginxLogging 		(Optional)

Logging defines logging related settings for NGINX.
nginxPlus
NginxPlus 		(Optional)

NginxPlus specifies NGINX Plus additional settings.
disableHTTP2
bool 		(Optional)

DisableHTTP2 defines if http2 should be disabled for all servers. If not specified, or set to false, http2 will be enabled for all servers.
useClusterIP
bool 		(Optional)

UseClusterIP configures NGINX to route to the Service ClusterIP and port instead of individual Pod IPs. When enabled, NGINX will target a single upstream server corresponding to the Service’s ClusterIP, which is useful for service mesh compatibility and other Kubernetes controllers/operators that require traffic to traverse the Service VIP. This setting applies only when the target Service has a ClusterIP. For headless Services (ClusterIP: None) and ExternalName Services, normal endpoint resolution is used instead. This setting is also not applied to L4/stream upstreams. A UseClusterIP value set in an UpstreamSettingsPolicy for a Service takes precedence over this setting. Defaults to false.
disableSNIHostValidation
bool 		(Optional)

DisableSNIHostValidation disables the validation that ensures the SNI hostname matches the Host header in HTTPS requests. When disabled, HTTPS connections can be reused for requests to different hostnames covered by the same certificate. This resolves HTTP/2 connection coalescing issues with wildcard certificates but introduces security risks as described in Gateway API GEP-3567. If not specified, defaults to false (validation enabled).
kubernetes
KubernetesSpec 		(Optional)

Kubernetes contains the configuration for the NGINX Deployment and Service Kubernetes objects.
workerConnections
int32 		(Optional)

WorkerConnections specifies the maximum number of simultaneous connections that can be opened by a worker process. Default is 1024.
workerProcesses
int32 		(Optional)

WorkerProcesses configures the number of NGINX worker processes. The value must be an integer between 1 and 1024. When unset, NGINX uses “auto” (one worker per CPU core).
workerRlimitNofile
int32 		(Optional)

WorkerRlimitNofile changes the limit on the maximum number of open files (RLIMIT_NOFILE) for worker processes. Used to raise the limit without restarting the main process. When unset, NGINX inherits the limit from the operating system. NGINX directive: https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile
dnsResolver
DNSResolver 		(Optional)

DNSResolver specifies the DNS resolver configuration for external name resolution. This enables support for routing to ExternalName Services.
serverTokens
string 		(Optional)

ServerTokens configures whether NGINX emits its version in the “Server” response header and on error pages.

OSS NGINX accepts: - “on”: Shows nginx and version (e.g. “nginx/1.25.0”) - “off”: Shows nginx only (e.g. “nginx”) - “build”: Shows version and build name (e.g. “nginx/1.25.0 (build-name)”)

NGINX Plus additionally accepts: - “”: Suppress the “Server” response header entirely - : Set a custom header value and supports variables

See: https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens NGINX directive: https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens Default is “off”.
compression
Compression 		(Optional)

Compression defines the configuration for HTTP response compression. When set, NGINX compresses responses for clients that support it, reducing bandwidth usage.
waf
WAFSpec 		(Optional)

WAF configures NGINX App Protect WAF functionality.
zoneSize
Size 		(Optional)

ZoneSize is the size of the shared memory zone used by the upstream. This memory zone is used to share the upstream configuration between nginx worker processes. The more servers that an upstream has, the larger memory zone is required. Default: OSS: 512k, Plus: 1m. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone
disableBaseHeaders
[]BaseHeaderName 		(Optional)

DisableBaseHeaders specifies which default X-* base headers should be omitted from being added to the base proxy_set_header directives in the NGINX configuration. This allows users to set these headers themselves without NGF overriding them.

Supported values are limited to X-* base headers and “”. A value of “” disables all X-* base headers.

ObservabilityPolicy

ObservabilityPolicy is a Direct Attached Policy. It provides a way to configure observability settings for the NGINX Gateway Fabric data plane. Used in conjunction with the NginxProxy CRD that is attached to the GatewayClass parametersRef.

Field Description
apiVersion
string		 gateway.nginx.org/v1alpha2
kind
string 		ObservabilityPolicy
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
ObservabilityPolicySpec

Spec defines the desired state of the ObservabilityPolicy.



tracing
Tracing 		(Optional)

Tracing allows for enabling and configuring tracing.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies the API object(s) to apply the policy to. Objects must be in the same namespace as the policy. Support: HTTPRoute, GRPCRoute.

TargetRefs must be distinct. This means that the multi-part key defined by kind and name must be unique across all targetRef entries in the ObservabilityPolicy.
status
sigs.k8s.io/gateway-api/apis/v1.PolicyStatus

Status defines the state of the ObservabilityPolicy.

AgentLogLevel (string alias)

(Appears on: NginxLogging)

AgentLevel defines the log level of the NGINX agent process.

Value Description

"debug"

AgentLogLevelDebug is the debug level NGINX agent logs.

"error"

AgentLogLevelError is the error level NGINX agent logs.

"fatal"

AgentLogLevelFatal is the fatal level NGINX agent logs.

"info"

AgentLogLevelInfo is the info level NGINX agent logs.

"panic"

AgentLogLevelPanic is the panic level NGINX agent logs.

AutoscalingSpec

(Appears on: DeploymentSpec)

AutoscalingSpec is the configuration for the Horizontal Pod Autoscaling.

Field Description
behavior
Kubernetes autoscaling/v2.HorizontalPodAutoscalerBehavior 		(Optional)

Behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively). If not set, the default HPAScalingRules for scale up and scale down are used.
targetCPUUtilizationPercentage
int32 		(Optional)

Target cpu utilization percentage of HPA.
targetMemoryUtilizationPercentage
int32 		(Optional)

Target memory utilization percentage of HPA.
minReplicas
int32 		(Optional)

Minimum number of replicas.
metrics
[]Kubernetes autoscaling/v2.MetricSpec 		(Optional)

Metrics configures additional metrics options.
maxReplicas
int32

Maximum number of replicas.
enable
bool

Enable or disable Horizontal Pod Autoscaler.

BaseHeaderName (string alias)

(Appears on: NginxProxySpec)

BaseHeaderName is the name of a base X-* header that can be disabled from being added to the base proxy_set_header directives in the NGINX configuration.

Value Description

"*"

AllXBaseHeaders disables all X-* base headers.

"X-Forwarded-For"

HeaderXForwardedFor is the X-Forwarded-For header.

"X-Forwarded-Host"

HeaderXForwardedHost is the X-Forwarded-Host header.

"X-Forwarded-Port"

HeaderXForwardedPort is the X-Forwarded-Port header.

"X-Forwarded-Proto"

HeaderXForwardedProto is the X-Forwarded-Proto header.

"X-Real-IP"

HeaderXRealIP is the X-Real-IP header.

Compression

(Appears on: NginxProxySpec)

Compression defines the configuration for HTTP response compression.

Field Description
gzip
GzipSettings 		(Optional)

Gzip defines gzip module-specific compression settings.
buffers
CompressionBuffers 		(Optional)

Buffers sets the number and size of buffers used to compress a response.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_buffers
level
int32 		(Optional)

Level sets the compression level. Higher values provide better compression but use more CPU.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_comp_level
minLength
int32 		(Optional)

MinLength sets the minimum length of a response that will be compressed. The length is determined from the “Content-Length” response header field.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_min_length
type
CompressionType

Type specifies the compression algorithm to use. Currently only gzip is supported.
mimeTypes
[]string 		(Optional)

MimeTypes specifies the MIME types to compress in addition to “text/html”. “text/html” is always compressed when compression is enabled. Wildcards like “text/*” are not supported by NGINX. Example: [“application/json”, “text/css”, “application/javascript”]

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_types

CompressionBuffers

(Appears on: Compression)

CompressionBuffers defines the number and size of buffers used for compression.

Field Description
size
Size

Size sets the size of each buffer.
number
int32

Number sets the number of buffers.

CompressionType (string alias)

(Appears on: Compression)

CompressionType defines the type of compression algorithm.

Value Description

"gzip"

GzipCompressionType specifies gzip compression.

ContainerSpec

(Appears on: DaemonSetSpec, DeploymentSpec)

ContainerSpec defines container fields for the NGINX container.

Field Description
debug
bool 		(Optional)

Debug enables debugging for NGINX by using the nginx-debug binary.
image
Image 		(Optional)

Image is the NGINX image to use.
resources
Kubernetes core/v1.ResourceRequirements 		(Optional)

Resources describes the compute resource requirements.
lifecycle
Kubernetes core/v1.Lifecycle 		(Optional)

Lifecycle describes actions that the management system should take in response to container lifecycle events. For the PostStart and PreStop lifecycle handlers, management of the container blocks until the action is complete, unless the container process fails, in which case the handler is aborted.
readinessProbe
ReadinessProbeSpec 		(Optional)

ReadinessProbe defines the readiness probe for the NGINX container.
hostPorts
[]HostPort 		(Optional)

HostPorts are the list of ports to expose on the host.
volumeMounts
[]Kubernetes core/v1.VolumeMount 		(Optional)

VolumeMounts describe the mounting of Volumes within a container.

DNSResolver

(Appears on: NginxProxySpec)

DNSResolver specifies the DNS resolver configuration for NGINX. This enables dynamic DNS resolution for ExternalName Services. Corresponds to the NGINX resolver directive: https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

Field Description
timeout
Duration 		(Optional)

Timeout specifies the timeout for name resolution.
cacheTTL
Duration 		(Optional)

CacheTTL specifies how long to cache DNS responses.
disableIPv6
bool 		(Optional)

DisableIPv6 disables IPv6 lookups. If not specified, or set to false, IPv6 lookups will be enabled.
addresses
[]DNSResolverAddress

Addresses specifies the list of DNS server addresses. Each address can be an IP address or hostname. Example: [{“type”: “IPAddress”, “value”: “8.8.8.8”}, {“type”: “Hostname”, “value”: “dns.google”}]

DNSResolverAddress

(Appears on: DNSResolver)

DNSResolverAddress specifies the address type and value for a DNS resolver address.

Field Description
type
DNSResolverAddressType

Type specifies the type of address.
value
string

Value specifies the address value. When Type is “IPAddress”, this must be a valid IPv4 or IPv6 address. When Type is “Hostname”, this must be a valid hostname.

DNSResolverAddressType (string alias)

(Appears on: DNSResolverAddress)

DNSResolverAddressType specifies the type of DNS resolver address.

Value Description

"Hostname"

DNSResolverHostnameType specifies that the address is a hostname.

"IPAddress"

DNSResolverIPAddressType specifies that the address is an IP address.

DaemonSetSpec

(Appears on: KubernetesSpec)

DaemonSet is the configuration for the NGINX DaemonSet.

Field Description
container
ContainerSpec 		(Optional)

Container defines container fields for the NGINX container.
wafContainers
WAFContainerSpec 		(Optional)

WAFContainers defines container specifications for NGINX App Protect WAF v5 containers. These containers are only deployed when WAF is enabled in the NginxProxy spec.
pod
PodSpec 		(Optional)

Pod defines Pod-specific fields.
patches
[]Patch 		(Optional)

Patches are custom patches to apply to the NGINX DaemonSet.

DeploymentSpec

(Appears on: KubernetesSpec)

Deployment is the configuration for the NGINX Deployment.

Field Description
replicas
int32 		(Optional)

Number of desired Pods.
autoscaling
AutoscalingSpec 		(Optional)

Autoscaling defines the configuration for Horizontal Pod Autoscaling.
podDisruptionBudget
PodDisruptionBudgetSpec 		(Optional)

PodDisruptionBudget is the configuration for limiting the number of concurrent disruptions of a pod. A PodDisruptionBudget is created when this field is set.
wafContainers
WAFContainerSpec 		(Optional)

WAFContainers defines container specifications for NGINX App Protect WAF v5 containers. These containers are only deployed when WAF is enabled in the NginxProxy spec.
pod
PodSpec 		(Optional)

Pod defines Pod-specific fields.
container
ContainerSpec 		(Optional)

Container defines container fields for the NGINX container.
patches
[]Patch 		(Optional)

Patches are custom patches to apply to the NGINX Deployment.

DisableTelemetryFeature (string alias)

(Appears on: Telemetry)

DisableTelemetryFeature is a telemetry feature that can be disabled.

Value Description

"DisableTracing"

DisableTracing disables the OpenTelemetry tracing feature.

ExternalTrafficPolicy (string alias)

(Appears on: ServiceSpec)

ExternalTrafficPolicy describes how nodes distribute service traffic they receive on one of the Service’s “externally-facing” addresses (NodePorts and LoadBalancer IPs).

Value Description

"Cluster"

ExternalTrafficPolicyCluster routes traffic to all endpoints.

"Local"

ExternalTrafficPolicyLocal preserves the source IP of the traffic by routing only to endpoints on the same node as the traffic was received on (dropping the traffic if there are no local endpoints).

GzipHTTPVersion (string alias)

(Appears on: GzipSettings)

GzipHTTPVersion defines the minimum HTTP version required for gzip compression.

Value Description

"1.0"

GzipHTTPVersion10 sets the minimum HTTP version to 1.0.

"1.1"

GzipHTTPVersion11 sets the minimum HTTP version to 1.1.

GzipProxiedType (string alias)

(Appears on: GzipSettings)

GzipProxiedType defines the conditions under which responses from proxied requests are compressed.

Value Description

"any"

GzipProxiedAny enables compression for all proxied requests.

"auth"

GzipProxiedAuth enables compression if a request header includes “Authorization”.

"expired"

GzipProxiedExpired enables compression if a response header includes the “Expires” field.

"no-cache"

GzipProxiedNoCache enables compression if a response header includes “Cache-Control” with the “no-cache” parameter.

"no_etag"

GzipProxiedNoETag enables compression if a response header does not include “ETag”.

"no_last_modified"

GzipProxiedNoLastModified enables compression if a response header does not include “Last-Modified”.

"no-store"

GzipProxiedNoStore enables compression if a response header includes “Cache-Control” with the “no-store” parameter.

"off"

GzipProxiedOff disables compression for all proxied requests.

"private"

GzipProxiedPrivate enables compression if a response header includes “Cache-Control” with the “private” parameter.

GzipSettings

(Appears on: Compression)

GzipSettings defines gzip module-specific compression settings.

Field Description
vary
bool 		(Optional)

Vary enables or disables inserting the “Vary: Accept-Encoding” response header when gzip compression is active.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_vary
httpVersion
GzipHTTPVersion 		(Optional)

HTTPVersion sets the minimum HTTP version of a request required to compress a response.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_http_version
disable
[]string 		(Optional)

Disable specifies regular expressions to match User-Agent headers of requests that should not be gzip-compressed.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_disable
proxied
[]GzipProxiedType 		(Optional)

Proxied enables or disables gzip compression for proxied requests depending on the request and response. Accepted values are: “off”, “expired”, “no-cache”, “no-store”, “private”, “no_last_modified”, “no_etag”, “auth”, “any”. Multiple values can be specified.

NGINX directive: https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_proxied

HostPort

(Appears on: ContainerSpec)

HostPort exposes an nginx container port on the host.

Field Description
port
int32

Port to expose on the host.
containerPort
int32

ContainerPort is the port on the nginx container to map to the HostPort.

IPFamilyType (string alias)

(Appears on: NginxProxySpec)

IPFamilyType specifies the IP family to be used by NGINX.

Value Description

"dual"

Dual specifies that NGINX will use both IPv4 and IPv6.

"ipv4"

IPv4 specifies that NGINX will use only IPv4.

"ipv6"

IPv6 specifies that NGINX will use only IPv6.

Image

(Appears on: ContainerSpec, WAFContainerConfig)

Image is the NGINX image to use.

Field Description
repository
string 		(Optional)

Repository is the image path. Default is ghcr.io/nginx/nginx-gateway-fabric/nginx.
tag
string 		(Optional)

Tag is the image tag to use. Default matches the tag of the control plane.
pullPolicy
PullPolicy 		(Optional)

PullPolicy describes a policy for if/when to pull a container image.

KubernetesSpec

(Appears on: NginxProxySpec)

KubernetesSpec contains the configuration for the NGINX Deployment and Service Kubernetes objects.

Field Description
deployment
DeploymentSpec 		(Optional)

Deployment is the configuration for the NGINX Deployment. This is the default deployment option.
daemonSet
DaemonSetSpec 		(Optional)

DaemonSet is the configuration for the NGINX DaemonSet.
service
ServiceSpec 		(Optional)

Service is the configuration for the NGINX Service.

Metrics

(Appears on: NginxProxySpec)

Metrics defines the configuration for Prometheus scraping metrics.

Field Description
port
int32 		(Optional)

Port where the Prometheus metrics are exposed.
disable
bool 		(Optional)

Disable serving Prometheus metrics on the listen port.

NginxAccessLog

(Appears on: NginxLogging)

NginxAccessLog defines the configuration for an NGINX access log.

Field Description
disable
bool 		(Optional)

Disable turns off access logging when set to true.
format
string 		(Optional)

Format specifies the custom log format string. If not specified, NGINX default ‘combined’ format is used. For now only path /dev/stdout can be used. Single quotes and line breaks are not allowed because the format is rendered inside a single-quoted NGINX log_format directive. See https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
escape
NginxAccessLogEscapeType 		(Optional)

Escape specifies how to escape characters in variables for access log. Possible values are: default, json, none. If not specified, ‘default’ escaping is used. See https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

NginxAccessLogEscapeType (string alias)

(Appears on: NginxAccessLog)

NginxAccessLogEscapeType defines the escape setting for variables in access log format.

Value Description

"default"

NginxAccessLogEscapeDefault specifies that characters ‘\“’, ‘\’, and other characters with values less than 32 or above 126 are escaped as ‘\xXX’.

"json"

NginxAccessLogEscapeJSON specifies that all characters not allowed in JSON strings are escaped. Characters ‘\“’ and ‘\’ are escaped as ‘\”’ and ‘\’, characters with values less than 32 are escaped as ‘\n’, ‘\r’, ‘\t’, ‘\b’, ‘\f’, or ‘\u00XX’.

"none"

NginxAccessLogEscapeNone disables escaping of characters.

NginxErrorLogFormat (string alias)

(Appears on: NginxLogging)

NginxErrorLogFormat defines the output format for NGINX error logs.

Value Description

"default"

NginxErrorLogFormatDefault uses NGINX’s standard error log format.

"json"

NginxErrorLogFormatJSON enables JSON-formatted error logs. Requires NGINX Plus.

NginxErrorLogLevel (string alias)

(Appears on: NginxLogging)

NginxErrorLogLevel type defines the log level of error logs for NGINX.

Value Description

"alert"

NginxLogLevelAlert is the alert level for NGINX error logs.

"crit"

NginxLogLevelCrit is the crit level for NGINX error logs.

"debug"

NginxLogLevelDebug is the debug level for NGINX error logs.

"emerg"

NginxLogLevelEmerg is the emerg level for NGINX error logs.

"error"

NginxLogLevelError is the error level for NGINX error logs.

"info"

NginxLogLevelInfo is the info level for NGINX error logs.

"notice"

NginxLogLevelNotice is the notice level for NGINX error logs.

"warn"

NginxLogLevelWarn is the warn level for NGINX error logs.

NginxLogging

(Appears on: NginxProxySpec)

NginxLogging defines logging related settings for NGINX.

Field Description
errorLevel
NginxErrorLogLevel 		(Optional)

ErrorLevel defines the error log level. Possible log levels listed in order of increasing severity are debug, info, notice, warn, error, crit, alert, and emerg. Setting a certain log level will cause all messages of the specified and more severe log levels to be logged. For example, the log level ‘error’ will cause error, crit, alert, and emerg messages to be logged. https://nginx.org/en/docs/ngx_core_module.html#error_log
errorLogFormat
NginxErrorLogFormat 		(Optional)

ErrorLogFormat controls the output format of the NGINX error_log directive. Set to ‘json’ to enable JSON-formatted error logs for NGINX Plus only and cannot be combined with errorLevel: debug. When set to ‘json’, NGINX Gateway Fabric also emits a JSON-formatted access log if the user has not supplied a custom access log format. See https://nginx.org/en/docs/ngx_core_module.html#error_log
agentLevel
AgentLogLevel 		(Optional)

AgentLevel defines the log level of the NGINX agent process. Changing this value results in a re-roll of the NGINX deployment.
accessLog
NginxAccessLog 		(Optional)

AccessLog defines the access log settings, including format itself and disabling option. For now only path /dev/stdout can be used.

NginxPlus

(Appears on: NginxProxySpec)

NginxPlus specifies NGINX Plus additional settings. These will only be applied if NGINX Plus is being used.

Field Description
allowedAddresses
[]NginxPlusAllowAddress 		(Optional)

AllowedAddresses specifies IPAddresses or CIDR blocks to the allow list for accessing the NGINX Plus API.

NginxPlusAllowAddress

(Appears on: NginxPlus)

NginxPlusAllowAddress specifies the address type and value for an NginxPlus allow address.

Field Description
type
NginxPlusAllowAddressType

Type specifies the type of address.
value
string

Value specifies the address value.

NginxPlusAllowAddressType (string alias)

(Appears on: NginxPlusAllowAddress)

NginxPlusAllowAddressType specifies the type of address.

Value Description

"CIDR"

NginxPlusAllowCIDRAddressType specifies that the address is a CIDR block.

"IPAddress"

NginxPlusAllowIPAddressType specifies that the address is an IP address.

NginxProxySpec

(Appears on: NginxProxy)

NginxProxySpec defines the desired state of the NginxProxy.

Field Description
ipFamily
IPFamilyType 		(Optional)

IPFamily specifies the IP family to be used by the NGINX. If not set, NGF inspects the default/kubernetes Service’s ipFamilies field at startup to obtain the IP family of the cluster and configure NGINX accordingly.
telemetry
Telemetry 		(Optional)

Telemetry specifies the OpenTelemetry configuration.
metrics
Metrics 		(Optional)

Metrics defines the configuration for Prometheus scraping metrics. Changing this value results in a re-roll of the NGINX deployment.
rewriteClientIP
RewriteClientIP 		(Optional)

RewriteClientIP defines configuration for rewriting the client IP to the original client’s IP.
logging
NginxLogging 		(Optional)

Logging defines logging related settings for NGINX.
nginxPlus
NginxPlus 		(Optional)

NginxPlus specifies NGINX Plus additional settings.
disableHTTP2
bool 		(Optional)

DisableHTTP2 defines if http2 should be disabled for all servers. If not specified, or set to false, http2 will be enabled for all servers.
useClusterIP
bool 		(Optional)

UseClusterIP configures NGINX to route to the Service ClusterIP and port instead of individual Pod IPs. When enabled, NGINX will target a single upstream server corresponding to the Service’s ClusterIP, which is useful for service mesh compatibility and other Kubernetes controllers/operators that require traffic to traverse the Service VIP. This setting applies only when the target Service has a ClusterIP. For headless Services (ClusterIP: None) and ExternalName Services, normal endpoint resolution is used instead. This setting is also not applied to L4/stream upstreams. A UseClusterIP value set in an UpstreamSettingsPolicy for a Service takes precedence over this setting. Defaults to false.
disableSNIHostValidation
bool 		(Optional)

DisableSNIHostValidation disables the validation that ensures the SNI hostname matches the Host header in HTTPS requests. When disabled, HTTPS connections can be reused for requests to different hostnames covered by the same certificate. This resolves HTTP/2 connection coalescing issues with wildcard certificates but introduces security risks as described in Gateway API GEP-3567. If not specified, defaults to false (validation enabled).
kubernetes
KubernetesSpec 		(Optional)

Kubernetes contains the configuration for the NGINX Deployment and Service Kubernetes objects.
workerConnections
int32 		(Optional)

WorkerConnections specifies the maximum number of simultaneous connections that can be opened by a worker process. Default is 1024.
workerProcesses
int32 		(Optional)

WorkerProcesses configures the number of NGINX worker processes. The value must be an integer between 1 and 1024. When unset, NGINX uses “auto” (one worker per CPU core).
workerRlimitNofile
int32 		(Optional)

WorkerRlimitNofile changes the limit on the maximum number of open files (RLIMIT_NOFILE) for worker processes. Used to raise the limit without restarting the main process. When unset, NGINX inherits the limit from the operating system. NGINX directive: https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile
dnsResolver
DNSResolver 		(Optional)

DNSResolver specifies the DNS resolver configuration for external name resolution. This enables support for routing to ExternalName Services.
serverTokens
string 		(Optional)

ServerTokens configures whether NGINX emits its version in the “Server” response header and on error pages.

OSS NGINX accepts: - “on”: Shows nginx and version (e.g. “nginx/1.25.0”) - “off”: Shows nginx only (e.g. “nginx”) - “build”: Shows version and build name (e.g. “nginx/1.25.0 (build-name)”)

NGINX Plus additionally accepts: - “”: Suppress the “Server” response header entirely - : Set a custom header value and supports variables

See: https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens NGINX directive: https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens Default is “off”.
compression
Compression 		(Optional)

Compression defines the configuration for HTTP response compression. When set, NGINX compresses responses for clients that support it, reducing bandwidth usage.
waf
WAFSpec 		(Optional)

WAF configures NGINX App Protect WAF functionality.
zoneSize
Size 		(Optional)

ZoneSize is the size of the shared memory zone used by the upstream. This memory zone is used to share the upstream configuration between nginx worker processes. The more servers that an upstream has, the larger memory zone is required. Default: OSS: 512k, Plus: 1m. Directive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone
disableBaseHeaders
[]BaseHeaderName 		(Optional)

DisableBaseHeaders specifies which default X-* base headers should be omitted from being added to the base proxy_set_header directives in the NGINX configuration. This allows users to set these headers themselves without NGF overriding them.

Supported values are limited to X-* base headers and “”. A value of “” disables all X-* base headers.

NodePort

(Appears on: ServiceSpec)

NodePort creates a port on each node on which the NGINX data plane service is exposed. The NodePort MUST map to a Gateway listener port, otherwise it will be ignored. If not specified, Kubernetes allocates a NodePort automatically if required. The default NodePort range enforced by Kubernetes is 30000-32767.

Field Description
port
int32

Port is the NodePort to expose.
listenerPort
int32

ListenerPort is the Gateway listener port that this NodePort maps to.

ObservabilityPolicySpec

(Appears on: ObservabilityPolicy)

ObservabilityPolicySpec defines the desired state of the ObservabilityPolicy.

Field Description
tracing
Tracing 		(Optional)

Tracing allows for enabling and configuring tracing.
targetRefs
[]sigs.k8s.io/gateway-api/apis/v1.LocalPolicyTargetReference

TargetRefs identifies the API object(s) to apply the policy to. Objects must be in the same namespace as the policy. Support: HTTPRoute, GRPCRoute.

TargetRefs must be distinct. This means that the multi-part key defined by kind and name must be unique across all targetRef entries in the ObservabilityPolicy.

Patch

(Appears on: DaemonSetSpec, DeploymentSpec, ServiceSpec)

Patch defines a patch to apply to a Kubernetes object.

Field Description
type
PatchType 		(Optional)

Type is the type of patch. Defaults to StrategicMerge.
value
k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.JSON 		(Optional)

Value is the patch data as raw JSON. For StrategicMerge and Merge patches, this should be a JSON object. For JSONPatch patches, this should be a JSON array of patch operations.

PatchType (string alias)

(Appears on: Patch)

PatchType specifies the type of patch.

Value Description

"JSONPatch"

PatchTypeJSONPatch uses JSON patch (RFC 6902).

"Merge"

PatchTypeMerge uses merge patch (RFC 7386).

"StrategicMerge"

PatchTypeStrategicMerge uses strategic merge patch.

PodDisruptionBudgetSpec

(Appears on: DeploymentSpec)

PodDisruptionBudgetSpec is the configuration for PodDisruptionBudget, which limits the number of concurrent disruptions of a pod.

Field Description
minAvailable
k8s.io/apimachinery/pkg/util/intstr.IntOrString 		(Optional)

MinAvailable is the minimum number of pods that must be available after an eviction. Value can be an absolute number (e.g. 1) or a percentage of desired pods (e.g. 50%). Mutually exclusive with MaxUnavailable.
maxUnavailable
k8s.io/apimachinery/pkg/util/intstr.IntOrString 		(Optional)

MaxUnavailable is the maximum number of pods that can be unavailable after an eviction. Value can be an absolute number (e.g. 1) or a percentage of desired pods (e.g. 50%). Mutually exclusive with MinAvailable.
unhealthyPodEvictionPolicy
Kubernetes policy/v1.UnhealthyPodEvictionPolicyType 		(Optional)

UnhealthyPodEvictionPolicy defines when unhealthy pods should be considered for eviction. Valid values are IfHealthyBudget and AlwaysAllow. Defaults to IfHealthyBudget if not set.

PodSpec

(Appears on: DaemonSetSpec, DeploymentSpec)

PodSpec defines Pod-specific fields.

Field Description
terminationGracePeriodSeconds
int64 		(Optional)

TerminationGracePeriodSeconds is the optional duration in seconds the pod needs to terminate gracefully. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.
affinity
Kubernetes core/v1.Affinity 		(Optional)

Affinity is the pod’s scheduling constraints.
nodeSelector
map[string]string 		(Optional)

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node.
tolerations
[]Kubernetes core/v1.Toleration 		(Optional)

Tolerations allow the scheduler to schedule Pods with matching taints.
volumes
[]Kubernetes core/v1.Volume 		(Optional)

Volumes represents named volumes in a pod that may be accessed by any container in the pod.
topologySpreadConstraints
[]Kubernetes core/v1.TopologySpreadConstraint 		(Optional)

TopologySpreadConstraints describes how a group of Pods ought to spread across topology domains. Scheduler will schedule Pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.

PullPolicy (string alias)

(Appears on: Image)

PullPolicy describes a policy for if/when to pull a container image.

Value Description

"Always"

PullAlways means that kubelet always attempts to pull the latest image. Container will fail if the pull fails.

"IfNotPresent"

PullIfNotPresent means that kubelet pulls if the image isn’t present on disk. Container will fail if the image isn’t present and the pull fails.

"Never"

PullNever means that kubelet never pulls an image, but only uses a local image. Container will fail if the image isn’t present.

ReadinessProbeSpec

(Appears on: ContainerSpec)

ReadinessProbeSpec defines the configuration for the NGINX readiness probe.

Field Description
port
int32 		(Optional)

Port is the port on which the readiness endpoint is exposed. If not specified, the default port is 8081.
path
string 		(Optional)

Path is the path on which the readiness endpoint is exposed. If not specified, the default path is /readyz. Must start with a forward slash and contain only valid URL path characters.
initialDelaySeconds
int32 		(Optional)

InitialDelaySeconds is the number of seconds after the container has started before the readiness probe is initiated. If not specified, the default is 3 seconds.
expose
bool 		(Optional)

Expose toggles whether the endpoint should be exposed through the Gateway Service object. This allows an external LoadBalancer to perform healthchecks. Default is false.

RewriteClientIP

(Appears on: NginxProxySpec)

RewriteClientIP specifies the configuration for rewriting the client’s IP address.

Field Description
mode
RewriteClientIPModeType 		(Optional)

Mode defines how NGINX will rewrite the client’s IP address. There are two possible modes: - ProxyProtocol: NGINX will rewrite the client’s IP using the PROXY protocol header. - XForwardedFor: NGINX will rewrite the client’s IP using the X-Forwarded-For header. Sets NGINX directive real_ip_header: https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
setIPRecursively
bool 		(Optional)

SetIPRecursively configures whether recursive search is used when selecting the client’s address from the X-Forwarded-For header. It is used in conjunction with TrustedAddresses. If enabled, NGINX will recurse on the values in X-Forwarded-Header from the end of array to start of array and select the first untrusted IP. For example, if X-Forwarded-For is [11.11.11.11, 22.22.22.22, 55.55.55.1], and TrustedAddresses is set to 55.55.55.1⁄32, NGINX will rewrite the client IP to 22.22.22.22. If disabled, NGINX will select the IP at the end of the array. In the previous example, 55.55.55.1 would be selected. Sets NGINX directive real_ip_recursive: https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive
trustedAddresses
[]RewriteClientIPAddress 		(Optional)

TrustedAddresses specifies the addresses that are trusted to send correct client IP information. If a request comes from a trusted address, NGINX will rewrite the client IP information, and forward it to the backend in the X-Forwarded-For* and X-Real-IP headers. If the request does not come from a trusted address, NGINX will not rewrite the client IP information. To trust all addresses (not recommended for production), set to 0.0.0.0/0. If no addresses are provided, NGINX will not rewrite the client IP information. Sets NGINX directive set_real_ip_from: https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from This field is required if mode is set.

RewriteClientIPAddress

(Appears on: RewriteClientIP)

RewriteClientIPAddress specifies the address type and value for a RewriteClientIP address.

Field Description
type
RewriteClientIPAddressType

Type specifies the type of address.
value
string

Value specifies the address value.

RewriteClientIPAddressType (string alias)

(Appears on: RewriteClientIPAddress)

RewriteClientIPAddressType specifies the type of address.

Value Description

"CIDR"

RewriteClientIPCIDRAddressType specifies that the address is a CIDR block.

"Hostname"

RewriteClientIPHostnameAddressType specifies that the address is a Hostname.

"IPAddress"

RewriteClientIPIPAddressType specifies that the address is an IP address.

RewriteClientIPModeType (string alias)

(Appears on: RewriteClientIP)

RewriteClientIPModeType defines how NGINX Gateway Fabric will determine the client’s original IP address.

Value Description

"ProxyProtocol"

RewriteClientIPModeProxyProtocol configures NGINX to accept PROXY protocol and set the client’s IP address to the IP address in the PROXY protocol header. Sets the proxy_protocol parameter on the listen directive of all servers and sets real_ip_header to proxy_protocol: https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header.

"XForwardedFor"

RewriteClientIPModeXForwardedFor configures NGINX to set the client’s IP address to the IP address in the X-Forwarded-For HTTP header. https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header.

ServiceSpec

(Appears on: KubernetesSpec)

ServiceSpec is the configuration for the NGINX Service.

Field Description
type
ServiceType 		(Optional)

ServiceType describes ingress method for the Service.
externalTrafficPolicy
ExternalTrafficPolicy 		(Optional)

ExternalTrafficPolicy describes how nodes distribute service traffic they receive on one of the Service’s “externally-facing” addresses (NodePorts and LoadBalancer IPs).
loadBalancerIP
string 		(Optional)

LoadBalancerIP is a static IP address for the load balancer. Requires service type to be LoadBalancer.
loadBalancerClass
string 		(Optional)

LoadBalancerClass is the class of the load balancer implementation this Service belongs to. Requires service type to be LoadBalancer.
loadBalancerSourceRanges
[]string 		(Optional)

LoadBalancerSourceRanges are the IP ranges (CIDR) that are allowed to access the load balancer. Requires service type to be LoadBalancer.
nodePorts
[]NodePort 		(Optional)

NodePorts are the list of NodePorts to expose on the NGINX data plane service. Each NodePort MUST map to a Gateway listener port, otherwise it will be ignored. The default NodePort range enforced by Kubernetes is 30000-32767.
patches
[]Patch 		(Optional)

Patches are custom patches to apply to the NGINX Service.

ServiceType (string alias)

(Appears on: ServiceSpec)

ServiceType describes ingress method for the Service.

Value Description

"ClusterIP"

ServiceTypeClusterIP means a Service will only be accessible inside the cluster, via the cluster IP.

"LoadBalancer"

ServiceTypeLoadBalancer means a Service will be exposed via an external load balancer (if the cloud provider supports it), in addition to ‘NodePort’ type.

"NodePort"

ServiceTypeNodePort means a Service will be exposed on one port of every node, in addition to ‘ClusterIP’ type.

Telemetry

(Appears on: NginxProxySpec)

Telemetry specifies the OpenTelemetry configuration.

Field Description
disabledFeatures
[]DisableTelemetryFeature 		(Optional)

DisabledFeatures specifies OpenTelemetry features to be disabled.
exporter
TelemetryExporter 		(Optional)

Exporter specifies OpenTelemetry export parameters.
serviceName
string 		(Optional)

ServiceName is the “service.name” attribute of the OpenTelemetry resource. Default is ‘ngf:gateway-namespace:gateway-name’. If a value is provided by the user, then the default becomes a prefix to that value.
spanAttributes
[]SpanAttribute 		(Optional)

SpanAttributes are custom key/value attributes that are added to each span.

TelemetryExporter

(Appears on: Telemetry)

TelemetryExporter specifies OpenTelemetry export parameters.

Field Description
interval
Duration 		(Optional)

Interval is the maximum interval between two exports. Default: https://nginx.org/en/docs/ngx_otel_module.html#otel_exporter
batchSize
int32 		(Optional)

BatchSize is the maximum number of spans to be sent in one batch per worker. Default: https://nginx.org/en/docs/ngx_otel_module.html#otel_exporter
batchCount
int32 		(Optional)

BatchCount is the number of pending batches per worker, spans exceeding the limit are dropped. Default: https://nginx.org/en/docs/ngx_otel_module.html#otel_exporter
endpoint
string 		(Optional)

Endpoint is the address of OTLP/gRPC endpoint that will accept telemetry data. Format: alphanumeric hostname with optional http scheme and optional port.

TraceContext (string alias)

(Appears on: Tracing)

TraceContext specifies how to propagate traceparent/tracestate headers.

Value Description

"extract"

TraceContextExtract uses an existing trace context from the request, so that the identifiers of a trace and the parent span are inherited from the incoming request.

"ignore"

TraceContextIgnore skips context headers processing.

"inject"

TraceContextInject adds a new context to the request, overwriting existing headers, if any.

"propagate"

TraceContextPropagate updates the existing context (combines extract and inject).

TraceStrategy (string alias)

(Appears on: Tracing)

TraceStrategy defines the tracing strategy.

Value Description

"parent"

TraceStrategyParent enables tracing and only records spans if the parent span was sampled.

"ratio"

TraceStrategyRatio enables ratio-based tracing, defaulting to 100% sampling rate.

Tracing

(Appears on: ObservabilityPolicySpec)

Tracing allows for enabling and configuring OpenTelemetry tracing.

Field Description
strategy
TraceStrategy

Strategy defines if tracing is ratio-based or parent-based.
ratio
int32 		(Optional)

Ratio is the percentage of traffic that should be sampled. Integer from 0 to 100. By default, 100% of http requests are traced. Not applicable for parent-based tracing. If ratio is set to 0, tracing is disabled.
context
TraceContext 		(Optional)

Context specifies how to propagate traceparent/tracestate headers. Default: https://nginx.org/en/docs/ngx_otel_module.html#otel_trace_context
spanName
string 		(Optional)

SpanName defines the name of the Otel span. By default is the name of the location for a request. If specified, applies to all locations that are created for a route. Format: must have all ‘“’ escaped and must not contain any ‘$’ or end with an unescaped ‘\’ Examples of invalid names: some-$value, quoted-“value”-name, unescaped
spanAttributes
[]SpanAttribute 		(Optional)

SpanAttributes are custom key/value attributes that are added to each span.

WAFContainerConfig

(Appears on: WAFContainerSpec)

WAFContainerConfig defines the configuration for a single WAF container.

Field Description
image
Image 		(Optional)

Image is the container image to use for this WAF container.
resources
Kubernetes core/v1.ResourceRequirements 		(Optional)

Resources describes the compute resource requirements for this WAF container.
volumeMounts
[]Kubernetes core/v1.VolumeMount 		(Optional)

VolumeMounts describe the mounting of Volumes within the WAF container.

WAFContainerSpec

(Appears on: DaemonSetSpec, DeploymentSpec)

WAFContainerSpec defines the container specifications for NGINX App Protect WAF v5. NAP v5 requires two additional containers: waf-enforcer and waf-config-mgr.

Field Description
enforcer
WAFContainerConfig 		(Optional)

Enforcer defines the configuration for the WAF enforcer container. This container performs the actual WAF enforcement and policy application.
configManager
WAFContainerConfig 		(Optional)

ConfigManager defines the configuration for the WAF configuration manager container. This container manages policy configuration and communication with the enforcer.

WAFSpec

(Appears on: NginxProxySpec)

WAFSpec configures NGINX App Protect WAF.

Field Description
enable
bool 		(Optional)

Enable enables NGINX App Protect WAF functionality. When enabled, NGINX Gateway Fabric will deploy additional WAF containers (waf-enforcer and waf-config-mgr) alongside the main NGINX container. Default is false.
disableCookieSeed
bool 		(Optional)

DisableCookieSeed disables the app_protect_cookie_seed directive. By default, NGF sets this directive to a stable value derived from the Gateway UID, ensuring WAF session cookies are consistent across multiple NGINX replicas. Set this to true if you have pre-compiled the cookie seed into your WAF policy bundles via the compiler global settings, to avoid conflicting with the compiled-in value. Default is false.
bundleFailOpen
bool 		(Optional)

BundleFailOpen controls the behavior when a WAF policy bundle (policy or log profile) has not yet been successfully fetched. When set to true, NGINX configuration is pushed and traffic is served without WAF protection until the bundle becomes available. When false (the default), the configuration push is withheld until the bundle is fetched, maintaining a fail-closed posture.

Generated with gen-crd-api-reference-docs

-------------------------------------------------------------------------------- title: "Command-line reference guide" url: https://docs.nginx.com/nginx-gateway-fabric/reference/cli-help/index.md -------------------------------------------------------------------------------- ## Overview Learn about the commands available for the executable file of the NGINX Gateway Fabric container. --- ## Controller This command runs the NGINX Gateway Fabric control plane. *Usage*: ```shell gateway controller [flags] ``` --- ### Flags | Name | Type | Description | |-------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | _gateway-ctlr-name_ | _string_ | The name of the Gateway controller. The controller name must be in the form: `DOMAIN/PATH`. The controller's domain is `gateway.nginx.org`. | | _gatewayclass_ | _string_ | The name of the GatewayClass resource. Every NGINX Gateway Fabric must have a unique corresponding GatewayClass resource. | | _nginx-plus_ | _bool_ | Enable support for NGINX Plus. | | _gateway-api-experimental-features_ | _bool_ | Enable the experimental features of Gateway API which are supported by NGINX Gateway Fabric. Requires the Gateway APIs installed from the experimental channel. | | _config_ | _string_ | The name of the NginxGateway resource to be used for this controller's dynamic configuration. Lives in the same namespace as the controller. | | _service_ | _string_ | The name of the service that fronts this NGINX Gateway Fabric pod. Lives in the same namespace as the controller. | | _metrics-disable_ | _bool_ | Disable exposing metrics in the Prometheus format (Default: `false`). | | _metrics-port_ | _int_ | Sets the port where the Prometheus metrics are exposed. An integer between 1024 - 65535 (Default: `9113`) | | _metrics-secure-serving_ | _bool_ | Configures if the metrics endpoint should be secured using https. Note that this endpoint will be secured with a self-signed certificate (Default `false`). | | _health-disable_ | _bool_ | Disable running the health probe server (Default: `false`). | | _health-port_ | _int_ | Set the port where the health probe server is exposed. An integer between 1024 - 65535 (Default: `8081`). | | _leader-election-disable_ | _bool_ | Disable leader election, which is used to avoid multiple replicas of the NGINX Gateway Fabric reporting the status of the Gateway API resources. If disabled, all replicas of NGINX Gateway Fabric will update the statuses of the Gateway API resources (Default: `false`). | | _leader-election-lock-name_ | _string_ | The name of the leader election lock. A lease object with this name will be created in the same namespace as the controller (Default: `"nginx-gateway-leader-election-lock"`). | | _leader-election-lease-duration_ | _duration_ | The duration that non-leader candidates will wait to force acquire leadership. Must be parsable by [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration). Must be greater than the renew deadline. If not set, defaults to controller-runtime's built-in default (`15s`). | | _leader-election-renew-deadline_ | _duration_ | The duration that the acting leader will retry refreshing leadership before giving up. Must be parsable by [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration). Must be greater than the retry period and less than the lease duration. Increase this value if leader election is failing due to slow Kubernetes API server responses. If not set, defaults to controller-runtime's built-in default (`10s`). | | _leader-election-retry-period_ | _duration_ | The duration the leader election clients should wait between action tries. Must be parsable by [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration). Must be less than the renew deadline. If not set, defaults to controller-runtime's built-in default (`2s`). | | _product-telemetry-disable_ | _bool_ | Disable the collection of product telemetry (Default: `false`). | | _nginx-docker-secret_ | _list_ | The name of the NGINX docker registry Secret(s). Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). | | _plm-storage-url_ | _string_ | The URL of the Policy Lifecycle Manager (PLM) storage service (HTTP or HTTPS). | | _plm-storage-credentials-secret_ | _string_ | The name of the Secret containing the PLM storage S3 secret access key (`seaweedfs_admin_secret`). Must exist in the same namespace as the NGINX Gateway Fabric control plane (default: `nginx-gateway`), unless prefixed with `/`. | | _plm-storage-ca-secret_ | _string_ | The name of the Secret containing the CA certificate (`ca.crt`) for verifying the PLM storage TLS server certificate. Must exist in the same namespace as the NGINX Gateway Fabric control plane (default: `nginx-gateway`), unless prefixed with `/`. | | _plm-storage-client-ssl-secret_ | _string_ | The name of the Secret containing the client certificate and key (`tls.crt`/`tls.key`) for mutual TLS with PLM storage. Must exist in the same namespace as the NGINX Gateway Fabric control plane (default: `nginx-gateway`), unless prefixed with `/`. | | _plm-storage-skip-verify_ | _bool_ | Disable TLS certificate verification for PLM storage connections. Use for testing only (Default: `false`). | | _usage-report-secret_ | _string_ | The name of the Secret containing the JWT for NGINX Plus usage reporting. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway) | | _usage-report-endpoint_ | _string_ | The endpoint of the NGINX Plus usage reporting server. | | _usage-report-resolver_ | _string_ | The nameserver used to resolve the NGINX Plus usage reporting endpoint. Used with NGINX Instance Manager. | | _usage-report-skip-verify_ | _bool_ | Disable client verification of the NGINX Plus usage reporting server certificate. | | _usage-report-ca-secret_ | _string_ | The name of the Secret containing the NGINX Instance Manager CA certificate. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway) | | _usage-report-client-ssl-secret_ | _string_ | The name of the Secret containing the client certificate and key for authenticating with NGINX Instance Manager. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway) | | _usage-report-enforce-initial-report_ | _bool_ | Enables or disables the 180-day grace period for sending the initial usage report. | | _snippets_ | _bool_ | Enable Snippets feature through SnippetsFilter and SnippetsPolicy APIs. SnippetsFilters allow inserting NGINX configuration into the generated NGINX config for HTTPRoute and GRPCRoute resources. SnippetsPolicies allow inserting NGINX configuration into the generated NGINX config for Gateway resources. | | _nginx-scc_ | _string_ | The name of the SecurityContextConstraints to be used with the NGINX data plane Pods. Only applicable in OpenShift. | | _nginx-one-dataplane-key-secret_ | _string_ | The name of the secret which holds the dataplane key that is required to authenticate with the NGINX One Console. Secret must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). | | _nginx-one-telemetry-endpoint-host_ | _string_ | The endpoint host that the NGINX One Console telemetry metrics will be sent to. | | _nginx-one-telemetry-endpoint-port_ | _int_ | The endpoint port that the NGINX One Console telemetry metrics will be sent to. | | _nginx-one-tls-skip-verify_ | _bool_ | Skip TLS verification for NGINX One Console connections. | | _nim-telemetry-endpoint-host_ | _string_ | The endpoint host that the NGINX Instance Manager telemetry metrics will be sent to. | | _nim-telemetry-endpoint-port_ | _int_ | The endpoint port that the NGINX Instance Manager telemetry metrics will be sent to. | | _gateway-api-inference-extension_ | _bool_ | Enable Gateway API Inference Extension support. Allows for configuring InferencePools to route traffic to AI workloads. (Default: `false`) | | _external-load-balancer_ | _bool_ | Enable ExternalLoadBalancer support. Allows for fronting a Gateway with an external load balancer. Supported load balancers: F5 BIG-IP, through F5 Container Ingress Services. (Default: `false`) | | _agent-tls-secret_ | _string_ | The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely communicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). (Default `agent-tls`) | | _endpoint-picker-disable-tls_ | _bool_ | Disables TLS when connecting to the EndpointPicker. Set to true only for development/testing or when using a service mesh for encryption. (Default: `false`) | | _endpoint-picker-tls-skip-verify_ | _bool_ | Disables server certificate verification when connecting to the EndpointPicker, if TLS is enabled. REQUIRED: Must be true until Gateway API Inference Extension EndpointPicker supports mounting certificates. (Default `true`) | | _watch-namespaces_ | _list_ | Comma-separated list of namespaces to watch for resources. If not set, all namespaces are watched. The controller's own namespace is always included. | | _payload-processor_ | _bool_ | Enable the PayloadProcessor API. PayloadProcessors enable declarative, ordered processing of HTTP request and response payloads by attaching to a Gateway or HTTPRoute, and are used to implement features such as Guardrails for AI workloads. | | _cluster-domain_ | _string_ | The DNS domain of your Kubernetes cluster. | ## Sleep This command sleeps for specified duration, then exits. _Usage_: ```shell gateway sleep [flags] ``` | Name | Type | Description | | -------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------- | | duration | `time.Duration` | Set the duration of sleep. Must be parsable by [`time.ParseDuration`](https://pkg.go.dev/time#ParseDuration). (default `30s`) | -------------------------------------------------------------------------------- title: "Configure tracing" url: https://docs.nginx.com/nginx-gateway-fabric/monitoring/tracing/index.md -------------------------------------------------------------------------------- This guide explains how to enable tracing on HTTPRoutes in NGINX Gateway Fabric using the OpenTelemetry Collector. Jaeger is used to process and collect the traces. ## Overview NGINX Gateway Fabric supports tracing using [OpenTelemetry](https://opentelemetry.io/). The official [NGINX OpenTelemetry Module](https://github.com/nginxinc/nginx-otel) instruments the NGINX data plane to export traces to a configured collector. Tracing data can be used with an OpenTelemetry Protocol (OTLP) exporter, such as the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector). This collector can then export data to one or more upstream collectors like [Jaeger](https://www.jaegertracing.io/), [DataDog](https://docs.datadoghq.com/tracing/), and many others. This is called the [Agent model](https://opentelemetry.io/docs/collector/deployment/agent/). ## Install the collectors The first step is to install the collectors. NGINX Gateway Fabric will be configured to export to the OpenTelemetry Collector, which is configured to export to Jaeger. This model allows the visualization collector (Jaeger) to be swapped with something else, or to add more collectors without needing to reconfigure NGINX Gateway Fabric. It is also possible to configure NGINX Gateway Fabric to export directly to Jaeger. Create the namespace: ```shell kubectl create namespace tracing ``` Download the following files containing the configurations for the collectors: - [Download: otel-collector.yaml](/ngf/otel-collector.yaml) - [Download: jaeger.yaml](/ngf/jaeger.yaml) **Note:** These collectors are for demonstration purposes and are not tuned for production use. Then install them: ```shell kubectl apply -f otel-collector.yaml -f jaeger.yaml -n tracing ``` Ensure the Pods are running: ```shell kubectl -n tracing get pods ``` ```text NAME READY STATUS RESTARTS AGE jaeger-8469f69b86-bfpk9 1/1 Running 0 9s otel-collector-f786b7dfd-h2x9l 1/1 Running 0 9s ``` Once running, you can access the Jaeger dashboard by using port-forwarding in the background: ```shell kubectl port-forward -n tracing svc/jaeger 16686:16686 & ``` Visit [http://127.0.0.1:16686](http://127.0.0.1:16686) to view the dashboard. ## Enable tracing To enable tracing, you must configure two resources: - `NginxProxy`: This resource contains global settings relating to the NGINX data plane. It is created and managed by the [cluster operator](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/), and is referenced in the `parametersRef` field of the GatewayClass. By default, an `NginxProxy` resource is created in the same namespace where NGINX Gateway Fabric is installed, attached to the GatewayClass. You can set configuration options in the `nginx` Helm value section, and the resource will be created and attached using the set values. When installed using the Helm chart, the NginxProxy resource is named `-proxy-config` and is created in the release Namespace. The `NginxProxy` resource contains configuration for the collector, and applies to all Gateways and routes under the GatewayClass. It does not enable tracing, but is a prerequisite to the next piece of configuration. **Note:** You can also override the tracing configuration for a particular Gateway by manually creating and attaching specific `NginxProxy` resources to target the different Gateways. This guide covers the global tracing configuration only. - `ObservabilityPolicy`: This resource is a [Direct PolicyAttachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that targets HTTPRoutes or GRPCRoutes. It is created by the [application developer](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/) and enables tracing for a specific route or routes. It requires the `NginxProxy` resource to exist in order to complete the tracing configuration. For all the possible configuration options for these resources, see the [API reference](/ngf/reference/api.md). ### Install NGINX Gateway Fabric with global tracing configuration **Note:** Ensure that you [install the Gateway API resources](/ngf/install/helm.md#installing-the-gateway-api-resources). Referencing the previously deployed collector, create the following `values.yaml` file for installing NGINX Gateway Fabric: ```yaml cat < values.yaml nginx: config: telemetry: exporter: endpoint: otel-collector.tracing.svc:4317 spanAttributes: - key: cluster-attribute-key value: cluster-attribute-value EOT ``` The span attribute will be added to all tracing spans. To install: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway -f values.yaml ``` You should see the following configuration: ```shell kubectl get nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway -o yaml ``` ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: ngf-proxy-config spec: telemetry: exporter: endpoint: otel-collector.tracing.svc:4317 spanAttributes: - key: cluster-attribute-key value: cluster-attribute-value ``` ```shell kubectl get gatewayclasses.gateway.networking.k8s.io nginx -o yaml ``` ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: nginx spec: controllerName: gateway.nginx.org/nginx-gateway-controller parametersRef: group: gateway.nginx.org kind: NginxProxy name: ngf-proxy-config status: conditions: - lastTransitionTime: "2024-05-22T15:18:35Z" message: GatewayClass is accepted observedGeneration: 1 reason: Accepted status: "True" type: Accepted - lastTransitionTime: "2024-05-22T15:18:35Z" message: Gateway API CRD versions are supported observedGeneration: 1 reason: SupportedVersion status: "True" type: SupportedVersion - lastTransitionTime: "2024-05-22T15:18:35Z" message: parametersRef resource is resolved observedGeneration: 1 reason: ResolvedRefs status: "True" type: ResolvedRefs ``` If you already have NGINX Gateway Fabric installed, then you can modify the `NginxProxy` resource to include the tracing configuration: ```shell kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway ``` You can now create the application, route, and tracing policy. ### Create the application and route Create the basic **coffee** application: ```yaml kubectl apply -f - < ``` Check that traffic can flow to the application. **Note:** If you have a DNS record allocated for `cafe.example.com`, you can send the request directly to that hostname, without needing to resolve. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` You should receive a response from the coffee Pod. ```text Server address: 10.244.0.69:8080 Server name: coffee-6b8b6d6486-k5w5w URI: /coffee ``` You shouldn't see any information from the [Jaeger dashboard](http://127.0.0.1:16686) yet: you need to create the `ObservabilityPolicy`. ### Create the ObservabilityPolicy To enable tracing for the coffee HTTPRoute, create the following policy: ```yaml kubectl apply -f - <:`. ![image](/ngf/img/jaeger-trace-overview.png)

Select a trace to view the attributes. ![image](/ngf/img/jaeger-trace-attributes.png) The trace includes the attribute from the global NginxProxy resource as well as the attribute from the ObservabilityPolicy. ## See also - [Data plane configuration](/ngf/how-to/data-plane-configuration.md): learn how to dynamically update the NGINX Gateway Fabric global data plane configuration, including how to override the telemetry configuration for a particular Gateway. - [Custom policies](/ngf/overview/custom-policies.md): learn about how NGINX Gateway Fabric custom policies work. - [API reference](/ngf/reference/api.md): all configuration fields for the policies mentioned in this guide -------------------------------------------------------------------------------- title: "F5 WAF for NGINX overview" url: https://docs.nginx.com/nginx-gateway-fabric/waf-integration/overview/index.md -------------------------------------------------------------------------------- F5 NGINX Gateway Fabric integrates with F5 WAF for NGINX to provide web application firewall protection. WAF policies are compiled externally and deployed to the data plane using the `WAFPolicy` custom resource. **Note:** F5 WAF for NGINX requires NGINX Plus and a separate F5 WAF for NGINX subscription. Contact your F5 sales representative for licensing details. --- ## Architecture F5 WAF for NGINX uses a multi-container architecture. When WAF is enabled, each NGINX Pod is extended with two sidecar containers: - **waf-enforcer**: Enforces WAF policies on incoming traffic. - **waf-config-mgr**: Manages WAF configuration and distributes policy bundles to the enforcer. Shared ephemeral volumes connect these containers to the main NGINX container. ```mermaid graph LR CP[NGINX Gateway Fabric] -->|gRPC: config + policy bundle| Agent subgraph NginxPod["NGINX Pod"] Agent[NGINX Agent] -->|writes bundle| Vol[(Shared Volume)] NGINX[NGINX + WAF Module] --- Vol Enforcer[WAF Enforcer] --- Vol ConfigMgr[WAF Config Mgr] --- Vol end Client[Client] ==> NGINX ==> Backend[Backend Service] ``` --- ## Enable WAF on the NginxProxy WAF is enabled by setting `waf.enable: true` on an `NginxProxy` resource. This instructs NGINX Gateway Fabric to deploy the WAF sidecar containers alongside the NGINX Pod. You can enable WAF at two levels: - **All Gateways** -- Set WAF on the GatewayClass-level `NginxProxy` so that every Gateway managed by this NGINX Gateway Fabric instance gets WAF sidecars by default. A per-Gateway `NginxProxy` can override this (for example, to disable WAF on a specific Gateway). - **Per Gateway** -- Create an `NginxProxy` and reference it from the `spec.infrastructure.parametersRef` field on a Gateway. Only that Gateway gets WAF sidecars. For details on how GatewayClass and Gateway-level NginxProxy settings are merged, see [Data plane configuration](/ngf/how-to/data-plane-configuration.md). ### Enable WAF for all Gateways To enable WAF at install time, use the **NGINX Plus with WAF** tab in the [Helm install guide](/ngf/install/helm.md). This sets the WAF-enabled NGINX Plus image (`nginx-plus-f5waf`) and enables WAF on the GatewayClass-level `NginxProxy`, so every Gateway gets WAF sidecars by default. To disable WAF for a specific Gateway, create a per-Gateway `NginxProxy` with `waf.enable: false` and reference it from that Gateway. **Note:** For additional WAF-related NginxProxy settings (including `disableCookieSeed`, `bundleFailOpen`, and custom WAF container images), see [Configure WAF settings](/ngf/waf-integration/configuration.md). ### Enable WAF per Gateway If you installed with the standard NGINX Plus image and want WAF on a specific Gateway only, create a per-Gateway `NginxProxy`. You must also set the NGINX image to `nginx-plus-f5waf`, because the standard `nginx-plus` image inherited from the GatewayClass doesn't include the WAF module: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: waf-enabled-proxy spec: waf: enable: true kubernetes: deployment: container: image: repository: private-registry.nginx.com/nginx-gateway-fabric/nginx-plus-f5waf ``` ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: gateway spec: gatewayClassName: nginx infrastructure: parametersRef: name: waf-enabled-proxy group: gateway.nginx.org kind: NginxProxy listeners: - name: http port: 80 protocol: HTTP ``` For the full list of available images, see [Supported container images](/ngf/overview/technical-specifications.md#supported-container-images). --- ## Policy lifecycle ### Bundles A WAF bundle is a compiled policy package produced by the [F5 WAF for NGINX compiler](/waf/configure/compiler.md). It contains the security policy, optional logging profile, [attack signatures](/waf/policies/attack-signatures.md), [threat campaign](/waf/policies/threat-campaigns.md) data, [bot signatures](/waf/policies/bot-signatures.md), and related metadata. The format lets the WAF engine load and enforce the policy at runtime. Pre-compiling policies into bundles results in faster, more reliable WAF startup: policies are resolved and validated at build time rather than on the running data plane. ### Compilation WAF policies must be compiled before they can be applied. Compilation takes a JSON policy definition (and optionally [global settings](/waf/configure/compiler.md) such as a cookie seed and [user-defined signatures](/waf/policies/user-signatures.md)) and produces a `.tgz` bundle. NGINX Gateway Fabric doesn't compile policies. Its role begins with fetching a compiled bundle and deploying it to the data plane. ### Source types Set the source type using the `spec.type` field on the `WAFPolicy` resource: | Type | Description | |--------|---------------------------------------------------------------------------------------------------| | `NIM` | NGINX Instance Manager -- fetched by policy name or UID using the NGINX Instance Manager API | | `N1C` | NGINX One Console -- fetched by policy name or object ID using the NGINX One Console API | | `HTTP` | Direct HTTP/HTTPS URL to a compiled bundle file | | `PLM` | Policy Lifecycle Management -- `APPolicy`/`APLogConf` CRDs, fetched from in-cluster storage | The `NIM`, `N1C`, and `HTTP` source types reference an externally compiled bundle through `policySource` (and `logSource` for log profiles). They detect updates by polling. The `PLM` source type is Kubernetes-native and event-driven: it references `APPolicy` and `APLogConf` custom resources through `policyRef` (and `logRef`), and doesn't require polling. See [PLM (Policy Lifecycle Management)](#plm-policy-lifecycle-management) below. For details on configuring each source type, see [Configure policy sources](/ngf/waf-integration/policy-sources.md). ### PLM (Policy Lifecycle Management) Policy Lifecycle Management (PLM) is a Kubernetes-native policy source. Instead of pointing NGINX Gateway Fabric at an externally compiled bundle, you define your WAF security posture as `APPolicy` and `APLogConf` custom resources in the cluster. The PLM controller watches these resources, compiles them automatically, and stores the resulting bundles in in-cluster S3-compatible storage. NGINX Gateway Fabric fetches the bundles from that storage and deploys them to the data plane. The following table summarizes how PLM differs from the HTTP, NGINX Instance Manager, and NGINX One Console source types: | Aspect | HTTP / NIM / N1C | PLM | |--------------------|-------------------------------------------------|---------------------------------------------------------| | Policy definition | Authored externally (file/Git, NIM, or N1C) | Authored in-cluster as `APPolicy`/`APLogConf` CRDs | | Compilation | External (compiler CLI/CI-CD, NIM, or N1C) | Automatic, by the PLM controller | | Bundle storage | HTTP server, NIM, or N1C | In-cluster S3-compatible storage | | `WAFPolicy` fields | `policySource` / `logSource` | `policyRef.apPolicyRef` / `logRef.apLogConfRef` | | Update detection | Polling (checksum or conditional GET) | Event-driven Kubernetes watch (no polling) | | Authentication | Per-`WAFPolicy` credentials Secret | Cluster-wide PLM storage credentials, set at install | | Network access | External egress to the policy source | Fully in-cluster | At runtime, the flow is: ```text Create APPolicy/APLogConf → PLM compiles and sets status.bundle.state: ready → NGINX Gateway Fabric detects the ready status via watch → fetches the bundle from in-cluster storage → deploys to the data plane ``` Changes to an `APPolicy` or `APLogConf` spec trigger recompilation and an automatic re-fetch. No polling is required, and no change to the `WAFPolicy` resource is needed. When a `WAFPolicy` references an `APPolicy` or `APLogConf` in a different namespace, create a [ReferenceGrant](https://gateway-api.sigs.k8s.io/api-types/referencegrant/) in the target namespace to permit the reference. **note:** PLM requires the PLM system to be installed in the cluster and PLM storage to be configured on NGINX Gateway Fabric at install time. For a complete walkthrough, see [Get started with F5 WAF for NGINX using PLM](/ngf/waf-integration/get-started-plm.md). --- ## Policy attachment `WAFPolicy` uses **inherited policy attachment**, following the [Gateway API policy attachment model](https://gateway-api.sigs.k8s.io/reference/policy-attachment/): - A **Gateway-level** `WAFPolicy` protects all HTTPRoutes and GRPCRoutes attached to that Gateway automatically. New routes inherit protection without any additional configuration. - A **Route-level** `WAFPolicy` can be applied to a specific HTTPRoute or GRPCRoute to override the Gateway-level policy for that route. - More specific (route-level) policies take precedence over less specific (gateway-level) policies. The route-level policy completely replaces the gateway-level policy for that route. There's no merging. - Only one `WAFPolicy` may target a given resource at a given level. If two policies target the same Gateway or Route, the second is rejected with `Accepted=False` and reason `Conflicted`. ```text Gateway-level WAFPolicy → HTTPRoute (inherited automatically) Route-level WAFPolicy → Overrides Gateway-level for that route only ``` **Note:** GRPCRoutes are protected in the same way as HTTPRoutes. To target a GRPCRoute, set `kind: GRPCRoute` in the `targetRefs` field. Built-in gRPC log profiles (`log_grpc_all`, `log_grpc_blocked`, `log_grpc_illegal`) are available for gRPC-specific security logging. --- ## See also - [Get started with F5 WAF for NGINX](/ngf/waf-integration/get-started-http.md) - [Get started with F5 WAF for NGINX using PLM](/ngf/waf-integration/get-started-plm.md) - [Configure policy sources (NGINX Instance Manager, NGINX One Console, and HTTP)](/ngf/waf-integration/policy-sources.md) - [Configure WAF settings](/ngf/waf-integration/configuration.md) - [WAFPolicy and NginxProxy API reference](/ngf/reference/api.md) - [F5 WAF for NGINX documentation](/waf/) -------------------------------------------------------------------------------- title: "Gateway architecture" url: https://docs.nginx.com/nginx-gateway-fabric/overview/gateway-architecture/index.md -------------------------------------------------------------------------------- Learn about the architecture and design principles of NGINX Gateway Fabric: a Kubernetes Gateway API implementation which uses NGINX as the data plane. This document is intended for: - _Cluster Operators_ who want to understand how NGINX Gateway Fabric works in production, how it manages traffic, and how to troubleshoot failures. - _Application Developers_ who would like to use NGINX Gateway Fabric to expose and route traffic to their applications within Kubernetes. The reader needs to be familiar with core Kubernetes concepts, such as pods, deployments, services, and endpoints. For an understanding of how NGINX itself works, you can read the ["Inside NGINX: How We Designed for Performance & Scale"](https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/) blog post. If you are interested in contributing to the project or learning about its internal implementation details, please see the [Developer Architecture Guide](https://github.com/nginx/nginx-gateway-fabric/tree/main/docs/architecture). --- ## NGINX Gateway Fabric Deployment Model and Architectural Overview NGINX Gateway Fabric splits its architecture into two main parts to provide better security, flexibility, and reliability: ### Control Plane: Centralized Management The control plane operates as a Deployment, serving as a [Kubernetes controller](https://kubernetes.io/docs/concepts/architecture/controller/) built with the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) library. It manages all aspects of resource provisioning and configuration for the NGINX data planes by watching Gateway API resources and other Kubernetes objects such as Services, Endpoints, and Secrets. Key functionalities include: - Dynamic provisioning: When a new Gateway resource is created, the control plane automatically provisions a dedicated NGINX Deployment and exposes it using a Service. - Configuration management: Kubernetes and Gateway API resources are translated into NGINX configurations, which are securely delivered to the data plane pods via a gRPC connection to the NGINX Agent. - Secure communication: By default, the gRPC connection uses self-signed certificates generated during installation. Integration with [cert-manager](https://cert-manager.io/) is also supported for optional certificate management. ### Data Plane: Autonomous Traffic Management Each NGINX data plane pod can be provisioned as an independent Deployment or DaemonSet containing an `nginx` container. This container runs both the `nginx` process and the [NGINX agent](https://github.com/nginx/agent), which is responsible for: - Applying configurations: The agent receives updates from the control plane and applies them to the NGINX instance. - Handling reloads: NGINX Agent handles configuration reconciliation and reloading NGINX, eliminating the need for shared volumes or Unix signals between the control plane and data plane pods. ### Gateway Resource Management Users can have multiple gateways running side-by-side in the same cluster. This supports flexible operation and isolation across Gateways: - Concurrent Gateways: Multiple Gateway objects can run simultaneously within a single installation. - 1:1 resource mapping: Each Gateway resource corresponds uniquely to a dedicated data plane deployment, ensuring clear delineation of ownership and operational segregation. --- ## High-level overview of NGINX Gateway Fabric in execution This figure depicts an example of NGINX Gateway Fabric exposing three web applications within a Kubernetes cluster to clients on the internet: ```mermaid {grid="wide"} graph LR %% Nodes and Relationships subgraph KubernetesCluster["Kubernetes Cluster"] subgraph NamespaceNGF["Namespace: nginx-gateway"] NGFControlPlanePod["NGINX Gateway Fabric Control Plane Pod"] NGFControlPlanePod --> KubernetesAPI["Kubernetes API"] end subgraph ApplicationsNamespaceA["Namespace: applications"] subgraph DataplaneComponentsA["Dataplane Components"] GatewayA["Gateway A
Listener: *.example.com"] subgraph NGINXDataPlanePodA["NGINX Data Plane Pod"] subgraph NGINXContainerA["NGINX Container"] NGINXProcessA["NGINX Process"] NGINXAgentA["NGINX Agent"] end end end subgraph HTTPRouteAAndApplications["HTTPRoutes and Applications"] HTTPRouteA["HTTPRoute A
Host: a.example.com"] HTTPRouteB["HTTPRoute B
Host: b.example.com"] ApplicationA["Application A
Pods: 2"] ApplicationB["Application B
Pods: 1"] end end subgraph ApplicationsNamespaceB["Namespace: applications-2"] subgraph DataplaneComponentsB["Dataplane Components"] GatewayB["Gateway B
Listener: *.other-example.com"] subgraph NGINXDataPlanePodB["NGINX Data Plane Pod"] subgraph NGINXContainerB["NGINX Container"] NGINXProcessB["NGINX Process"] NGINXAgentB["NGINX Agent"] end end end subgraph HTTPRouteBandApplications["HTTPRoutes and Applications"] HTTPRouteC["HTTPRoute C
Host: c.other-example.com"] ApplicationC["Application C
Pods: 1"] end end end subgraph UsersAndClients["Users and Clients"] UserOperator["Cluster Operator"] UserDevA["Application Developer A"] UserDevB["Application Developer B"] ClientA["Client A"] ClientB["Client B"] end subgraph SharedInfrastructure["Public Endpoint"] PublicEndpoint["TCP Load Balancer / NodePort"] end %% Updated Traffic Flow ClientA-->|a.example.com|PublicEndpoint ClientB-->|c.other-example.com|PublicEndpoint PublicEndpoint==>NGINXProcessA PublicEndpoint==>NGINXProcessB NGINXProcessA==>ApplicationA NGINXProcessA==>ApplicationB NGINXProcessB==>ApplicationC %% Kubernetes Configuration Flow HTTPRouteA-->GatewayA HTTPRouteB-->GatewayA HTTPRouteC-->GatewayB UserOperator-->KubernetesAPI NGFControlPlanePod--gRPC-->NGINXAgentA NGFControlPlanePod--gRPC-->NGINXAgentB NGINXAgentA-->NGINXProcessA NGINXAgentB-->NGINXProcessB UserDevA-->KubernetesAPI UserDevB-->KubernetesAPI %% Styling style UserOperator fill:#66CDAA,stroke:#333,stroke-width:2px style GatewayA fill:#66CDAA,stroke:#333,stroke-width:2px style GatewayB fill:#66CDAA,stroke:#333,stroke-width:2px style NGFControlPlanePod fill:#66CDAA,stroke:#333,stroke-width:2px style NGINXProcessA fill:#66CDAA,stroke:#333,stroke-width:2px style NGINXProcessB fill:#66CDAA,stroke:#333,stroke-width:2px style KubernetesAPI fill:#9370DB,stroke:#333,stroke-width:2px style HTTPRouteAAndApplications fill:#E0FFFF,stroke:#333,stroke-width:2px style HTTPRouteBandApplications fill:#E0FFFF,stroke:#333,stroke-width:2px style UserDevA fill:#FFA07A,stroke:#333,stroke-width:2px style HTTPRouteA fill:#FFA07A,stroke:#333,stroke-width:2px style HTTPRouteB fill:#FFA07A,stroke:#333,stroke-width:2px style ApplicationA fill:#FFA07A,stroke:#333,stroke-width:2px style ApplicationB fill:#FFA07A,stroke:#333,stroke-width:2px style ClientA fill:#FFA07A,stroke:#333,stroke-width:2px style UserDevB fill:#87CEEB,stroke:#333,stroke-width:2px style HTTPRouteC fill:#87CEEB,stroke:#333,stroke-width:2px style ApplicationC fill:#87CEEB,stroke:#333,stroke-width:2px style ClientB fill:#87CEEB,stroke:#333,stroke-width:2px style PublicEndpoint fill:#FFD700,stroke:#333,stroke-width:2px ``` **Note:** The figure does not show many of the necessary Kubernetes resources the Cluster Operators and Application Developers need to create, like deployment and services. The figure shows: | **Category** | **Description** | |-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| | **Namespaces** | - _Namespace: nginx-gateway_: Contains the NGINX Gateway Fabric Control Plane Pod, responsible for managing Gateway API configurations and provisioning NGINX Data Plane Pods.
- _Namespace: applications_: Contains Gateway A for `*.example.com`, handling Application A and Application B.
- _Namespace: applications-2_: Contains Gateway B for `*.other-example.com`, handling Application C. | | **Users** | - _Cluster Operator_: Sets up the NGINX Gateway Fabric Control Plane Pod and manages Gateway API resources by provisioning Gateways (A and B).
- _Developers A & B_: Developers deploy their applications and create HTTPRoutes associated with their Gateways. | | **Clients** | - _Client A_: Interacts with Application A through `a.example.com`.
- _Client B_: Interacts with Application C through `c.other-example.com`. | | **NGINX Gateway Fabric Control Plane Pod** | The control plane pod, deployed in the `nginx-gateway` namespace, communicates with the Kubernetes API to:
- Fetch Gateway API resources.
- Dynamically provision and configure NGINX Data Plane Pods.
- Deliver traffic routing and configuration updates to NGINX Agent instances over gRPC. | | **Gateways** | - _Gateway A_: Listens for requests under `*.example.com`. Routes:
   • _HTTPRoute A_: Routes requests to `a.example.com` into Application A.
   • _HTTPRoute B_: Routes requests to `b.example.com` into Application B.
- _Gateway B_: Listens for requests under `*.other-example.com`. Routes:
   • _HTTPRoute C_: Routes requests to `c.other-example.com` into Application C. | | **Applications** | - _Application A_: Deployed by Developer A (2 pods), routed by Gateway A via HTTPRoute A.
- _Application B_: Deployed by Developer A (1 pod), routed by Gateway A via HTTPRoute B.
- _Application C_: Deployed by Developer B (1 pod), routed by Gateway B via HTTPRoute C. | | **NGINX Data Plane Pods** | - _NGINX Data Plane Pod A_: Handles traffic routed from Gateway A:
   • _NGINX Process A_: Forwards requests to Application A and Application B as defined in Gateway A's HTTPRoutes.
   • _NGINX Agent A_: Receives configuration updates from the NGINX Gateway Fabric Control Plane Pod via gRPC.
- _NGINX Data Plane Pod B_: Manages traffic routed from Gateway B:
   • _NGINX Process B_: Forwards requests to Application C as defined in Gateway B’s HTTPRoute.
   • _NGINX Agent B_: Receives configuration updates via gRPC from the NGINX Gateway Fabric Control Plane Pod. | | **Traffic Flow** | - _Client A_:
   1. Sends requests to `a.example.com` via the Public Endpoint.
   2. Requests are routed by Gateway A and processed by NGINX Process A.
   3. Traffic is forwarded to Application A.
- _Client B_:
   1. Sends requests to `c.other-example.com` via the Public Endpoint.
   2. Requests are routed by Gateway B and processed by NGINX Process B.
   3. Traffic is forwarded to Application C. | | **Public Endpoint** | A shared entry point (TCP Load Balancer or NodePort) that exposes the NGINX Data Plane externally to forward client traffic into the cluster. | | **Kubernetes API** | Acts as the central hub for resource management:
- Fetches Gateway API resources for Gateway A and Gateway B.
- Facilitates NGINX configuration updates via the NGINX Gateway Fabric Control Plane Pod. | _Color Coding_ : - Cluster Operator resources (e.g., NGINX Gateway Fabric, NGINX Pods and Gateways) are marked in _green_. - Resources owned by _Application Developer A_ (e.g., HTTPRoute A, Application A) are marked in _orange_. - Resources owned by _Application Developer B_ (e.g., HTTPRoute B, Application C) are marked in _blue_. --- ## NGINX Gateway Fabric: Component Communication Workflow ```mermaid graph LR %% Main Components subgraph nginx-gateway["Namespace: nginx-gateway"] NGFPod[NGINX Gateway Fabric Control Plane Pod] end subgraph NGINXDataPlane["NGINX Data Plane Pod"] NGINXAgent[NGINX Agent] NGINXMaster[NGINX Master] NGINXWorker[NGINX Worker] ConfigFiles[Config Files] end F5Telemetry[F5 Telemetry Service] PrometheusMonitor[Prometheus] KubernetesAPI[Kubernetes API] Client[Client] BackendApplication[Backend Application] %% High-Level Configuration Flow KubernetesAPI -->|"(1) Updates Resources"| NGFPod NGFPod -->|"(2) Sends Configuration Metadata via gRPC"| NGINXAgent NGINXAgent -->|"(3) Validates & Writes Configuration"| ConfigFiles NGINXAgent -->|"(4) Signals NGINX Master to Reload"| NGINXMaster %% Prometheus Monitoring PrometheusMonitor -->|"(5) Fetches Metrics from NGINX"| NGINXWorker %% Telemetry Data NGFPod -->|"(6) Sends Telemetry Data"| F5Telemetry %% Client Traffic Flow Client -->|"(7) Sends Traffic"| NGINXWorker NGINXWorker -->|"(8) Routes Traffic"| BackendApplication %% Styling classDef important fill:#66CDAA,stroke:#333,stroke-width:2px; classDef metrics fill:#FFC0CB,stroke:#333,stroke-width:2px; classDef io fill:#FFD700,stroke:#333,stroke-width:2px; class KubernetesAPI,NGFPod important; class PrometheusMonitor,F5Telemetry metrics; class NGINXAgent,NGINXMaster,NGINXWorker,ConfigFiles io; class Client,BackendApplication important; ``` The following table describes the connections, preceeded by their types in parentheses. For brevity, the suffix "process" has been omitted from the process descriptions. | # | Component/Protocol | Description | | ---| ----------------------- | ------------------------------------------------------------------------------------------------------------ | | 1 | Kubernetes API (HTTPS) | _Kubernetes API → NGINX Gateway Fabric Control Plane Pod_: The NGINX Gateway Fabric Control Plane Pod (in the `nginx-gateway` namespace) watches the Kubernetes API for updates to Gateway API resources (e.g., Gateways, HTTPRoutes), fetching the latest configuration to manage routing and traffic control. | | 2 | gRPC | _NGINX Gateway Fabric Control Plane Pod → NGINX Agent_: The NGINX Gateway Fabric Control Plane Pod processes Gateway API resources, generates NGINX configuration settings, and securely delivers them to the NGINX Agent inside the NGINX Data Plane Pod via gRPC. | | 3 | File I/O | _NGINX Agent → Config Files_: The NGINX Agent (within the NGINX Data Plane Pod) validates the configuration metadata received from the Control Plane Pod and writes it to NGINX configuration files within the pod. These files store dynamic routing rules and traffic settings. | | 4 | Signal | _NGINX Agent → NGINX Master_: After writing the configuration files, the NGINX Agent signals the NGINX Master process to reload the configuration. This ensures the NGINX Data Plane Pod immediately applies the updated routes and settings. | | 5 | HTTP/HTTPS | _Prometheus → NGINX Worker_: Prometheus collects runtime metrics (e.g., traffic statistics, request rates, and active connections) from the NGINX Worker process, which is part of the NGINX Data Plane Pod. The `/metrics` endpoint exposes these metrics for monitoring and observability. | | 6 | HTTPS | _NGINX Gateway Fabric Control Plane Pod → F5 Telemetry Service_: The NGINX Gateway Fabric Control Plane Pod sends telemetry data (e.g., API requests handled, usage metrics, performance stats, error rates) to the external F5 Telemetry Service for centralized monitoring and diagnostics. | | 7 | HTTP/HTTPS | _Client → NGINX Worker_: Clients send incoming application traffic (e.g., HTTP/HTTPS requests) to the NGINX Worker process within the NGINX Data Plane Pod. These requests are typically routed through a shared Public Endpoint (e.g., LoadBalancer or NodePort) before reaching the NGINX Data Plane. | | 8 | HTTP/HTTPS | _NGINX Worker → Backend Application_: The NGINX Worker process forwards client traffic to the appropriate backend application services (e.g., Pods) as defined in the routing rules and configuration received from the Control Plane Pod. | --- ### Additional features and enhancements when using NGINX Plus NGINX Gateway Fabric supports both NGINX Open Source and NGINX Plus. While the previous diagram shows NGINX Open Source, using NGINX Plus provides additional capabilities, including: - The ability for administrators to connect to the NGINX Plus API on port 8765 (restricted to localhost by default). - Dynamic updates to upstream servers without requiring a full reload: - Changes to upstream servers, such as application scaling (e.g., adding or removing pods in Kubernetes), can be applied using the [NGINX Plus API](http://nginx.org/en/docs/http/ngx_http_api_module.html). - This reduces the frequency of configuration reloads, minimizing potential disruptions and improving system stability during updates. These features enable reduced downtime, improved performance during scaling events, and more fine-grained control over traffic management. --- ### Resilience and fault isolation This architecture separates the control plane and data plane, creating clear operational boundaries that improve resilience and fault isolation: #### Control plane resilience In the event of a control plane failure or downtime: - Existing data plane pods continue serving traffic using their last-valid cached configurations. - Updates to routes or Gateways are temporarily paused, but stable traffic delivery continues without degradation. - Recovery restores functionality, resynchronizing configuration updates seamlessly. #### Data plane resilience If a data plane pod encounters an outage or restarts: - Only routes tied to the specific linked Gateway object experience brief disruptions. - Configurations automatically resynchronize with the data plane upon pod restart, minimizing the scope of impact. - Other data plane pods remain unaffected and continue serving traffic normally. --- ## Pod readiness The control plane (`nginx-gateway`) and data plane (`nginx`) containers provide a readiness endpoint at `/readyz`. A [readiness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes) periodically checks this endpoint during startup. The probe reports `200 OK` when: - The control plane is ready to configure the NGINX data planes. - The data plane is ready to handle traffic. This marks the pods ready ensuring traffic is routed to healthy pods, ensuring reliable startup and smooth operations. -------------------------------------------------------------------------------- title: "Use Manifests to install NGINX Gateway Fabric with NGINX Open Source" url: https://docs.nginx.com/nginx-gateway-fabric/install/manifests/open-source/index.md -------------------------------------------------------------------------------- This page describes how to use Manifests to install NGINX Gateway Fabric with NGINX Open Source. It explains how to install the Gateway API resources and add authentication certificates, then deploy NGINX Gateway Fabric and its custom resource definitions. By following these instructions, you will finish with a functional NGINX Gateway Fabric instance for your Kubernetes cluster. **Note:** To learn which Gateway API resources NGINX Gateway Fabric currently supports, view the [Gateway API Compatibility](/ngf/overview/gateway-api-compatibility.md) topic. To install an experimental NGINX Gateway Fabric version view the [Use Manifests to install NGINX Gateway Fabric (experimental) with NGINX Open Source](/ngf/install/manifests/open-source-experimental.md) topic. ## Before you begin To complete this guide, you will need the following pre-requisites: - [A supported Kubernetes version](/ngf/overview/technical-specifications.md) - A functional Kubernetes cluster ## Install the Gateway API resources **Note:** If you have already installed Gateway API resources in your cluster, ensure they are a version [supported by NGINX Gateway Fabric](/ngf/overview/technical-specifications.md) To install the Gateway API resources, use `kubectl kustomize`: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl apply -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created ``` You should also create the _nginx-gateway_ namespace, which is used by the Manifest files by default: ```shell kubectl create namespace nginx-gateway ``` ## Install cert-manager Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ## Add certificates for secure authentication **Note:** These steps use a self-signed issuer, which should not be used in production environments. For production environments, you should use a real [CA issuer](https://cert-manager.io/docs/configuration/ca/). First, create a CA (certificate authority) issuer: ```yaml kubectl apply -f - <= v1.18.0, the default value changed from `Never` to `Always`. certificate.cert-manager.io/nginx-gateway-ca created issuer.cert-manager.io/nginx-gateway-issuer created ``` You will then need to create a server certificate for the NGINX Gateway Fabric control plane (server): **Note:** The default service name is _nginx-gateway_, and the namespace is _nginx-gateway_, so the `dnsNames` value should be `nginx-gateway.nginx-gateway.svc`. This value becomes the name of the NGINX Gateway Fabric control plane service. ```yaml {hl_lines=[13]} kubectl apply -f - <coffee] pod1[Pod
coffee] pod2[Pod
coffee] end svc --> pod1 & pod2 class pod1,pod2,svc appNode classDef appNode fill:#edbd8c,stroke:#D9822B ``` Using this architecture, the **coffee** application is not accessible outside the cluster. We want to expose this application on the hostname "cafe.example.com" so that clients outside the cluster can access it. Install NGINX Gateway Fabric and create two Gateway API resources: a [gateway](https://gateway-api.sigs.k8s.io/reference/spec/#gateway) and an [HTTPRoute](https://gateway-api.sigs.k8s.io/reference/spec/#httproute). Using these resources we will configure a simple routing rule to match all HTTP traffic with the hostname "cafe.example.com" and route it to the **coffee** service. --- ## Set up Create the **coffee** application in Kubernetes by copying and pasting the following block into your terminal: ```yaml kubectl apply -f - < 80/TCP 77s ``` --- ## Application architecture with NGINX Gateway Fabric To route traffic to the **coffee** application, we will create a Gateway and HTTPRoute. The following diagram shows the configuration we are creating in the next step: ```mermaid graph LR subgraph config [Namespace default] subgraph padding [" "] direction LR style config fill:#FFFFFF,stroke:#000000 subgraph gw[Gateway cafe] subgraph gwPadding [" "] gwContents[HTTP/80] end end subgraph hr[HTTPRoute coffee] subgraph hrPadding [" "] hrContents[cafe.example.com] subgraph describeMatchAll [Match all
traffic] subgraph describeMatchPadding [" "] matchAll[Host: *
Path: *] end end subgraph describeService [Group matching
pods within a Service] subgraph describePadding [" "] coffeeSvc[Service
coffee] end end end end end end gwContents --> hrContents --> matchAll --> coffeeSvc class padding,gwPadding,hrPadding,describeMatchAll,describeService,describePadding,describeMatchPadding noBorder class gw gateway class hr httpRoute class matchAll,hrContents,coffeeSvc appDevNode class gwContents clusterOppNode classDef noBorder stroke:none,fill:none,text-align:center classDef default fill:#FFFFFF,stroke:#000000 classDef gateway fill:#FFFFFF,stroke:#blue,stroke-dasharray: 3 3,text-align:center classDef httpRoute fill:#FFFFFF,stroke:#D9822B,stroke-dasharray: 3 3,text-align:center classDef appDevNode fill:#edbd8c,stroke:#D9822B classDef clusterOppNode fill:lightblue,stroke:darkblue ``` We need a Gateway to create an entry point for HTTP traffic coming into the cluster. The **cafe** Gateway we are going to create will open an entry point to the cluster on port 80 for HTTP traffic. To route HTTP traffic from the Gateway to the **coffee** service, we need to create an HTTPRoute named **coffee** and attach it to the Gateway. This HTTPRoute will have a single routing rule that routes all traffic to the hostname "cafe.example.com" from the Gateway to the **coffee** service. Once NGINX Gateway Fabric processes the **cafe** Gateway and **coffee** HTTPRoute, it will configure a data plane (NGINX) to route all HTTP requests sent to "cafe.example.com" to the pods that the **coffee** service targets: ```mermaid graph LR style cluster fill:#FFFFFF,stroke:#000000 clients[Clients] ngfSvc["Public IP Address for
cafe.example.com"] subgraph cluster [Kubernetes Cluster] subgraph appNs [Namespace
default] subgraph nsPadding [" "] nginxPod[Pod NGINX] coffeePod1[Pod coffee] coffeePod2[Pod coffee] end end end ngfSvc --> nginxPod nginxPod --> coffeePod1 & coffeePod2 clients --> ngfSvc class clusterPadding,nsPadding,clusterPadding2 noBorder class gwNS,appNs namespace class nginxPod nginxNode class coffeePod1,coffeePod2 coffeeNode class ngfSvc svc class clients clientNode classDef noBorder stroke:none,fill:none classDef default fill:#FFFFFF,stroke:#000000 classDef namespace fill:#FFFFFF,stroke:#036ffc,stroke-dasharray: 5 5,text-align:center classDef nginxNode fill:#b4e0ad,stroke:#2AA317 classDef svc fill:lightblue,stroke:darkblue classDef coffeeNode fill:#edbd8c,stroke:#D9822B classDef clientNode fill:#D3D3D3 ``` The **coffee** service is omitted from the diagram above because the NGINX Pod routes directly to the pods that the **coffee** service targets. **Note:** In the diagrams above, all resources that are the responsibility of the cluster operator are shown in blue. The orange resources are the responsibility of the application developers. See the [roles and personas](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#roles-and-personas_1) Gateway API document for more information on these roles. --- ## Create the Gateway API resources To create the **cafe** gateway, copy and paste the following into your terminal: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. This Gateway is associated with NGINX Gateway Fabric through the **gatewayClassName** field. The default installation of NGINX Gateway Fabric creates a GatewayClass with the name **nginx**. NGINX Gateway Fabric will only configure Gateways with a **gatewayClassName** of **nginx** unless you change the name via the `--gatewayclass` [command-line flag](/ngf/reference/cli-help.md#controller). We specify a [listener](https://gateway-api.sigs.k8s.io/reference/spec/#listener) on the Gateway to open an entry point on the cluster. In this case, since the coffee application accepts HTTP requests, we create an HTTP listener, named **http**, that listens on port 80. By default, Gateways only allow routes (such as HTTPRoutes) to attach if they are in the same namespace as the Gateway. If you want to change this behavior, you can set the [**allowedRoutes**](https://gateway-api.sigs.k8s.io/reference/spec/#allowedroutes) field. Next you will create the HTTPRoute by copying and pasting the following into your terminal: ```yaml kubectl apply -f - < 404 Not Found

404 Not Found

nginx/1.25.2
``` --- ## Upgrade WebSocket connections NGINX Gateway Fabric can upgrade HTTP/1.1 requests to websocket connections when the appropriate headers are present and the backend supports websocket connections. The backend is responsible for completing the handshake by responding with `101 Switching Protocols`. To ensure this example works, your backend must support websocket connections. Send a `curl` request that includes the required websocket upgrade headers, described below: - `Connection: Upgrade` — requests a protocol change on the current connection. - `Upgrade: websocket` — indicates the desired protocol. - `Sec-WebSocket-Version: 13` — required websocket protocol version. - `Sec-WebSocket-Key` — a random, base64-encoded nonce used by the server to compute `Sec-WebSocket-Accept`. ```shell curl --http1.1 --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -H "Upgrade: websocket" -H "Connection: upgrade" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: $(openssl rand -base64 16)" ``` A websocket-capable backend will reply with: ```text < HTTP/1.1 101 Switching Protocols < Server: nginx < Connection: upgrade < Upgrade: websocket < Sec-WebSocket-Accept: On5gtTNGCqB4Emnh7Ck4T0b2gks= ``` Receiving a `101 Switching Protocols` response along with the `Upgrade`, `Connection`, and `Sec-WebSocket-Accept` headers confirms that the websocket connection upgrade completed successfully. --- ## Troubleshooting If you have any issues while testing the configuration, try the following to debug your configuration and setup: - Make sure you set the shell variables $GW_IP and $GW_PORT to the public IP and port of the NGINX Service. Refer to the [Installation](/ngf/install/) guides for more information. - Check the status of the gateway: ```shell kubectl describe gateway cafe ``` The gateway status should look similar to this: ```text Status: Addresses: Type: IPAddress Value: 10.244.0.85 Conditions: Last Transition Time: 2023-08-15T20:57:21Z Message: Gateway is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2023-08-15T20:57:21Z Message: Gateway is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed Listeners: Attached Routes: 1 Conditions: Last Transition Time: 2023-08-15T20:57:21Z Message: Listener is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2023-08-15T20:57:21Z Message: Listener is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed Last Transition Time: 2023-08-15T20:57:21Z Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Last Transition Time: 2023-08-15T20:57:21Z Message: No conflicts Observed Generation: 1 Reason: NoConflicts Status: False Type: Conflicted Name: http ``` Check that the conditions match and that the attached routes for the **http** listener equals 1. If it is 0, there may be an issue with the HTTPRoute. - Check the status of the HTTPRoute: ```shell kubectl describe httproute coffee ``` The HTTPRoute status should look similar to this: ```text Status: Parents: Conditions: Last Transition Time: 2023-08-15T20:57:21Z Message: The route is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2023-08-15T20:57:21Z Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Controller Name: gateway.nginx.org/nginx-gateway-controller Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: cafe Namespace: default ``` Check for any error messages in the conditions. - Check the generated nginx config: ```shell kubectl exec -it deployments/cafe-nginx -- nginx -T ``` The config should contain a server block with the server name "cafe.example.com" that listens on port 80. This server block should have a single location `/` that proxy passes to the coffee upstream: ```nginx configuration server { listen 80; server_name cafe.example.com; location / { ... proxy_pass http://default_coffee_80$request_uri; # the upstream is named default_coffee_80 ... } } ``` There should also be an upstream block with a name that matches the upstream in the **proxy_pass** directive. This upstream block should contain the pod IPs of the **coffee** pods: ```nginx configuration upstream default_coffee_80 { ... server 10.12.0.18:8080; # these should be the pod IPs of the coffee pods server 10.12.0.19:8080; ... } ``` **Note:** The entire configuration is not shown because it is subject to change. Ellipses indicate that there's configuration not shown. If your issue persists, [contact us](https://github.com/nginx/nginx-gateway-fabric#contacts). --- ## See also To learn more about the Gateway API and the resources we created in this guide, check out the following resources: - [Gateway API Overview](https://gateway-api.sigs.k8s.io/concepts/api-overview/) - [Deploying a simple Gateway](https://gateway-api.sigs.k8s.io/guides/getting-started/simple-gateway/) - [HTTP Routing](https://gateway-api.sigs.k8s.io/guides/http-routing/) -------------------------------------------------------------------------------- title: "Secure traffic using Let's Encrypt and cert-manager" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-security/integrate-cert-manager/index.md -------------------------------------------------------------------------------- Learn how to issue and manage certificates using Let's Encrypt and cert-manager. ## Overview Securing client server communication is a crucial part of modern application architectures. One of the most important steps in this process is implementing HTTPS (HTTP over TLS/SSL) for all communications. This encrypts the data transmitted between the client and server, preventing eavesdropping and tampering. To do this, you need an SSL/TLS certificate from a trusted Certificate Authority (CA). However, issuing and managing certificates can be a complicated manual process. Luckily, there are many services and tools available to simplify and automate certificate issuance and management. Follow the steps in this guide to: - Configure HTTPS for your application using a [gateway](https://gateway-api.sigs.k8s.io/api-types/gateway/). - Use [Let’s Encrypt](https://letsencrypt.org) as the Certificate Authority (CA) issuing the TLS certificate. - Use [cert-manager](https://cert-manager.io) to automate the provisioning and management of the certificate. --- ## Before you begin You need: - Administrator access to a Kubernetes cluster. - [Helm](https://helm.sh) and [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) must be installed locally. - [NGINX Gateway Fabric deployed](/ngf/install/) in the Kubernetes cluster. - A DNS-resolvable domain name is required. It must resolve to the public endpoint of the NGINX Gateway Fabric deployment, and this public endpoint must be an external IP address or alias accessible over the internet. The process here will depend on your DNS provider. This DNS name will need to be resolvable from the Let’s Encrypt servers, which may require that you wait for the record to propagate before it will work. --- ## Secure traffic using Let's Encrypt and cert-manager ![cert-manager ACME challenge and certificate management with Gateway API](ngf/img/cert-manager-gateway-workflow.png) The diagram above shows a simplified representation of the cert-manager ACME challenge and certificate issuance process using Gateway API. Please note that not all of the kubernetes objects created in this process are represented in this diagram. At a high level, the process looks like this: 1. We deploy cert-manager and create a ClusterIssuer which specifies Let’s Encrypt as our CA and gateway as our ACME HTTP01 challenge solver. 1. We create a gateway resource for our domain (cafe.example.com) and configure cert-manager integration using an annotation. 1. This starts the certificate issuance process – cert-manager contacts Let’s Encrypt to obtain a certificate, and Let’s Encrypt starts the ACME challenge. As part of this challenge, cert-manager creates a temporary HTTPRoute resource which directs the traffic through NGINX Gateway Fabric to verify we control the domain name in the certificate request. 1. Once the domain has been verified, the certificate is issued. Cert-manager stores the keypair in a Kubernetes secret that is referenced by the gateway resource. As a result, NGINX is configured to terminate HTTPS traffic from clients using this signed keypair. 1. We deploy our application and our HTTPRoute which defines our routing rules. The routing rules defined configure NGINX to direct requests to `https://cafe.example.com/coffee` to our coffee-app application, and to use the HTTPS listener defined in our gateway resource. 1. When the client connects to `https://cafe.example.com/coffee`, the request is routed to the coffee-app application and the communication is secured using the signed keypair contained in the cafe-secret secret. 1. The certificate will be automatically renewed when it is close to expiry, the secret will be updated using the new certificate, and NGINX Gateway Fabric will dynamically update the keypair on the filesystem used by NGINX for HTTPS termination once the secret is updated. --- ## Securing traffic ### Deploy cert-manager Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` --- ### Create a ClusterIssuer Next we need to create a [ClusterIssuer](https://cert-manager.io/docs/concepts/issuer/), a Kubernetes resource that represents the certificate authority (CA) that will generate the signed certificates by honouring certificate signing requests. We are using the ACME Issuer type, and Let's Encrypt as the CA server. In order for Let's Encypt to verify that we own the domain a certificate is being requested for, we must complete "challenges". This is to ensure clients are unable to request certificates for domains they do not own. We will configure the issuer to use a HTTP01 challenge, and our gateway resource that we will create in the next step as the solver. To read more about HTTP01 challenges, see the [cert-manager documentation](https://cert-manager.io/docs/configuration/acme/http01/). Use the following YAML definition to create the resource, but please note the `email` field must be updated to your own email address. ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email: my-name@example.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: # Secret resource that will be used to store the account's private key. name: issuer-account-key # Add a single challenge solver, HTTP01 using NGINX Gateway Fabric solvers: - http01: gatewayHTTPRoute: parentRefs: # This is the name of the Gateway that will be created in the next step - name: gateway namespace: default kind: Gateway ``` --- ### Deploy our Gateway with the cert-manager annotation Next we need to deploy our gateway. You can use the YAML manifest below, updating the `spec.listeners[1].hostname` field to the required value for your environment. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: gateway annotations: # This is the name of the ClusterIssuer created in the previous step cert-manager.io/cluster-issuer: letsencrypt-prod spec: gatewayClassName: nginx listeners: - name: http port: 80 protocol: HTTP - name: https # Important: The hostname needs to be set to your domain hostname: "cafe.example.com" port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - kind: Secret name: cafe-secret ``` It's worth noting a couple of key details in this manifest: - The cert-manager annotation is present in the metadata – this enables the cert-manager integration, and tells cert-manager which ClusterIssuer configuration it should use for the certificates. - There are two listeners configured, an HTTP listener on port 80, and an HTTPS listener on port 443. - The HTTP listener on port 80 is required for the HTTP01 ACME challenge to work. This is because as part of the HTTP01 challenge, a temporary HTTPRoute will be created by cert-manager to solve the ACME challenge, and this HTTPRoute requires a listener on port 80. See the [HTTP01 Gateway API solver documentation](https://cert-manager.io/docs/configuration/acme/http01/#configuring-the-http-01-gateway-api-solver) for more information. - The HTTPS listener on port 443 is the listener we will use in our HTTPRoute in the next step. Cert-manager will create a certificate for this listener block. - The hostname needs to set to the required value. A new certificate will be issued from the `letsencrypt-prod` ClusterIssuer for the domain, e.g. "cafe.example.com", once the ACME challenge is successful. Once the certificate has been issued, cert-manager will create a certificate resource on the cluster and the `cafe-secret` Secret containing the signed keypair in the same Namespace as the gateway. We can verify the secret has been created successfully using `kubectl`. Note it will take a little bit of time for the challenge to complete and the secret to be created: ```shell kubectl get secret cafe-secret ``` ```text NAME TYPE DATA AGE cafe-secret kubernetes.io/tls 2 20s ``` --- - Multiple `certificateRefs` apply to the listener's `hostname` and are not automatically selected based on the hostnames of attached HTTPRoute resources. Use separate HTTPS listeners for different domains that require different certificates. ### Deploy our application and HTTPRoute Now we can create our coffee deployment and service, and configure the routing rules. You can use the following manifest to create the deployment and service: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: coffee spec: replicas: 1 selector: matchLabels: app: coffee template: metadata: labels: app: coffee spec: containers: - name: coffee image: nginxdemos/nginx-hello:plain-text ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: coffee spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: coffee ``` Deploy our HTTPRoute to configure our routing rules for the coffee application. Note the `parentRefs` section in the spec refers to the listener configured in the previous step. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: coffee spec: parentRefs: - name: gateway sectionName: https hostnames: # Update the hostname to match what is configured in the Gateway resource - "cafe.example.com" rules: - matches: - path: type: PathPrefix value: /coffee backendRefs: - name: coffee port: 80 ``` --- ## Testing To test everything has worked correctly, we can use curl to the navigate to our endpoint, for example, `https://cafe.example.com/coffee`. To verify using curl, we can use the `-v` option to increase verbosity and inspect the presented certificate. ```shell curl https://cafe.example.com/coffee -v ``` The output will look similar to this: ```text * Trying 54.195.47.105:443... * Connected to cafe.example.com (54.195.47.105) port 443 (#0) * ALPN: offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 * ALPN: server accepted http/1.1 * Server certificate: * subject: CN=cafe.example.com * start date: Aug 11 08:22:11 2023 GMT * expire date: Nov 9 08:22:10 2023 GMT * subjectAltName: host "cafe.example.com" matched cert's "cafe.example.com" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * using HTTP/1.1 > GET /coffee HTTP/1.1 > Host: cafe.example.com > User-Agent: curl/7.88.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.25.1 < Date: Fri, 11 Aug 2023 10:03:21 GMT < Content-Type: text/plain < Content-Length: 163 < Connection: keep-alive < Expires: Fri, 11 Aug 2023 10:03:20 GMT < Cache-Control: no-cache < Server address: 192.168.78.136:8080 Server name: coffee-9bf875848-xvkqv Date: 11/Aug/2023:10:03:21 +0000 URI: /coffee Request ID: e64c54a2ac253375ac085d48980f000a * Connection #0 to host cafe.example.com left intact ``` --- ## Troubleshooting - To troubleshoot any issues related to the cert-manager installation or issuer setup, see [the cert-manager troubleshooting guide](https://cert-manager.io/docs/troubleshooting/). - To troubleshoot the HTTP01 ACME challenge, please see the cert-manager [ACME troubleshooting guide](https://cert-manager.io/docs/troubleshooting/acme/). - Note that for the HTTP01 challenge to work using the gateway resource, HTTPS redirect must not be configured. - The temporary HTTPRoute created by cert-manager routes the traffic between cert-manager and the Let's Encrypt server through NGINX Gateway Fabric. If the challenge is not successful, it may be useful to inspect the NGINX logs to see the ACME challenge requests. You should see something like the following: ```shell kubectl logs -n <...> 52.208.162.19 - - [15/Aug/2023:13:18:12 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "cert-manager-challenges/v1.12.0 (linux/amd64) cert-manager/bd192c4f76dd883f9ee908035b894ffb49002384" 52.208.162.19 - - [15/Aug/2023:13:18:14 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "cert-manager-challenges/v1.12.0 (linux/amd64) cert-manager/bd192c4f76dd883f9ee908035b894ffb49002384" 52.208.162.19 - - [15/Aug/2023:13:18:16 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "cert-manager-challenges/v1.12.0 (linux/amd64) cert-manager/bd192c4f76dd883f9ee908035b894ffb49002384" 52.208.162.19 - - [15/Aug/2023:13:18:18 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "cert-manager-challenges/v1.12.0 (linux/amd64) cert-manager/bd192c4f76dd883f9ee908035b894ffb49002384" 52.208.162.19 - - [15/Aug/2023:13:18:20 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "cert-manager-challenges/v1.12.0 (linux/amd64) cert-manager/bd192c4f76dd883f9ee908035b894ffb49002384" 3.128.204.81 - - [15/Aug/2023:13:18:22 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 23.178.112.204 - - [15/Aug/2023:13:18:22 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 35.166.192.222 - - [15/Aug/2023:13:18:22 +0000] "GET /.well-known/acme-challenge/bXQn27Lenax2AJKmOOS523T-MWOKeFhL0bvrouNkUc4 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" <...> ``` ## HTTPS Traffic without SNI (Server Name Indication) Some frontend load balancers strip out SNI information before the traffic reaches the NGINX gateway. In order for NGINX to still process and forward this traffic properly, you must define your HTTPS Listener without a hostname. This instructs NGINX Gateway Fabric to configure a default HTTPS virtual server to handle non-SNI traffic. The TLS configuration on this Listener will be used to verify and terminate TLS for this traffic, before the Host header is then used to forward to the proper virtual server to handle the request. You can attach your HTTPRoutes to this empty Listener. By default, NGINX Gateway Fabric verifies that the Listener hostname matches both the SNI and Host header on an incoming client request. This does not require the SNI and Host header to be the same. This is to avoid misdirected requests, and returns a 421 response code. If you run into issues and want to disable this SNI/Host verification, you can update the [NginxProxy CRD](/ngf/how-to/data-plane-configuration.md) with the following field in the spec: ```yaml spec: disableSNIHostValidation: true ``` --- ## See also - [Gateway docs](https://gateway-api.sigs.k8s.io) - [Cert-manager gateway usage](https://cert-manager.io/docs/usage/gateway/) - [Cert-manager ACME](https://cert-manager.io/docs/configuration/acme/) - [Let’s Encrypt](https://letsencrypt.org) - [NGINX HTTPS docs](https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/) -------------------------------------------------------------------------------- title: "Use F5 BIG-IP as an external load balancer" url: https://docs.nginx.com/nginx-gateway-fabric/external-loadbalancers/gateway-link/quickstart/index.md description: Configure an ExternalLoadBalancer so F5 BIG-IP acts as the external load balancer for an NGINX Gateway Fabric Gateway in a single cluster, preserving client IP addresses with the PROXY protocol. -------------------------------------------------------------------------------- This guide describes how to use an F5 BIG-IP system as the external load balancer for an NGINX Gateway Fabric Gateway. ## Overview GatewayLink integrates NGINX Gateway Fabric with F5 BIG-IP Container Ingress Services to configure an F5 BIG-IP system as the external load balancer for a Gateway. You describe the desired BIG-IP configuration through the `ExternalLoadBalancer` custom resource. In this guide, the F5 IPAM Controller allocates the address that BIG-IP listens on, and an iRule preserves the original client address by forwarding it to NGINX using the PROXY protocol. ### How configuration reaches BIG-IP NGINX Gateway Fabric watches `ExternalLoadBalancer` resources. For each one, it creates an `IngressLink` resource, the custom resource F5 Container Ingress Services uses to describe a Gateway that BIG-IP fronts. The IngressLink carries the settings from the ExternalLoadBalancer spec, along with a label selector that matches the Gateway's data plane Service. F5 Container Ingress Services watches IngressLink resources. It resolves the selector to the data plane Service, reads its node addresses and NodePorts, and compiles them into an AS3 declaration. It posts that declaration to the AS3 endpoint on BIG-IP, which creates the virtual server and its pool. F5 Container Ingress Services reposts the declaration whenever the endpoints or the IngressLink change, so BIG-IP stays current as Pods come and go. ```mermaid flowchart LR A[ExternalLoadBalancer
NGINX Gateway Fabric] --> B[IngressLink
F5 Container Ingress Services] B --> C[AS3 declaration
POST to BIG-IP] C --> D[BIG-IP
virtual server, pool] ``` ## Before you begin You need: - A Kubernetes cluster. - An F5 BIG-IP system running version or later, and an account on it with administrator privileges. - Network access from the cluster to the BIG-IP system, and from BIG-IP to the cluster node addresses. - Python 3.14 or later. This guide installs the AS3 extension, the F5 IPAM Controller, F5 Container Ingress Services, and NGINX Gateway Fabric. The shell commands in this guide read the following environment variables, so set them once in the shell you work from and the commands can be copied as they appear: ```shell export BIGIP_ADDRESS="192.0.2.10:443" export BIGIP_USERNAME="admin" export BIGIP_PASSWORD="" export IPAM_ADDRESS_RANGE="192.0.2.100-192.0.2.110" ``` - `BIGIP_ADDRESS` is the BIG-IP management address, including the port. BIG-IP listens on 443 by default. - `BIGIP_USERNAME` and `BIGIP_PASSWORD` are your BIG-IP credentials. - `IPAM_ADDRESS_RANGE` is a free address range on the BIG-IP subnet, which the F5 IPAM Controller allocates from. You choose this range in [Install the F5 IPAM Controller](#install-the-f5-ipam-controller). Two more variables are set later, once their values exist: - `ALLOCATED_ADDRESS` is the virtual server address the F5 IPAM Controller allocates, read from the `IngressLink` status in [Verify the configuration](#verify-the-configuration). - `NGINX_POD_NAME` is the name of an NGINX Pod, used when reading its logs. ## Prepare BIG-IP In this section you install the AS3 extension and create the two BIG-IP objects this guide depends on: a partition for F5 Container Ingress Services to own, and an iRule that adds a PROXY protocol header. ### AS3 extension F5 Container Ingress Services configures BIG-IP by posting AS3 declarations, so AS3 must be installed before anything else. Follow [Downloading and installing the BIG-IP AS3 package](https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/installation.html) in the F5 documentation, then return here. ### Partition Create a partition named `k8s` for F5 Container Ingress Services to own: ```shell curl -sku "$BIGIP_USERNAME:$BIGIP_PASSWORD" -X POST "https://$BIGIP_ADDRESS/mgmt/tm/auth/partition" \ -H "Content-Type: application/json" -d '{"name":"k8s"}' ``` The response describes the new partition: ```json { "name": "k8s", "fullPath": "k8s", "defaultRouteDomain": 0 } ``` F5 Container Ingress Services manages the full contents of its partition. The partition cannot be `Common`, because Container Ingress Services must not modify shared configuration. ### TCP iRule This guide uses a TCP iRule named `Proxy_Protocol_iRule`: ```text when SERVER_CONNECTED { TCP::respond "PROXY TCP[IP::version] [IP::client_addr] [clientside {IP::local_addr}] [TCP::client_port] [clientside {TCP::local_port}]\r\n" } ``` The iRule runs on the `SERVER_CONNECTED` event, which fires when BIG-IP opens a connection to NGINX, before any application data is sent. It writes a single PROXY protocol header onto that connection. The header carries the original client address, so NGINX can report it instead of the BIG-IP self-IP address. To create the iRule: ```shell curl -sku "$BIGIP_USERNAME:$BIGIP_PASSWORD" -X POST "https://$BIGIP_ADDRESS/mgmt/tm/ltm/rule" \ -H "Content-Type: application/json" -d '{ "name": "Proxy_Protocol_iRule", "apiAnonymous": "when SERVER_CONNECTED {\n TCP::respond \"PROXY TCP[IP::version] [IP::client_addr] [clientside {IP::local_addr}] [TCP::client_port] [clientside {TCP::local_port}]\\r\\n\"\n}" }' ``` The response describes the new iRule: ```json { "name": "Proxy_Protocol_iRule", "fullPath": "/Common/Proxy_Protocol_iRule", "apiAnonymous": "when SERVER_CONNECTED { ... }" } ``` ## Install F5 Container Ingress Services ### Install the F5 IPAM Controller The F5 IPAM Controller allocates the virtual server address from a range you define, so you do not have to pick and track an address by hand. Allocation is a handoff between the two controllers through a shared `IPAM` resource. F5 Container Ingress Services creates that resource on startup when it is installed with `--ipam=true`. When an `IngressLink` names an IPAM label, Container Ingress Services adds an entry to the resource `spec` requesting an address under that label. The F5 IPAM Controller watches the same resource, takes an address from the range configured for that label, and records the assignment in the resource `status`. Container Ingress Services reads the address from the status and uses it as the virtual server address in the AS3 declaration. Install the F5 IPAM Controller before Container Ingress Services, so it is watching by the time the first request is made. Install the `IPAM` custom resource definition: ```yaml kubectl apply -f - < ``` F5 Container Ingress Services writes this status after posting the AS3 declaration. A status of `OK` means BIG-IP accepted the declaration, and `vsAddress` is the address the F5 IPAM Controller allocated. Store that address for the remaining commands: ```shell export ALLOCATED_ADDRESS=$(kubectl get ingresslink gateway-nginx -o jsonpath='{.status.vsAddress}') ``` Send a request through BIG-IP: ```shell curl -H "Host: cafe.example.com" http://$ALLOCATED_ADDRESS/coffee ``` The request returns `200 OK` with a response body from the backend application. ```text Server address: 10.42.0.43:8080 Server name: coffee-7b9578cff9-t7r7v Date: 05/Aug/2026:14:33:40 +0000 URI: /coffee Request ID: a2ae0944885fdf99bb5f86038aeae84f ``` Confirm NGINX sees the original client address: ```shell export NGINX_POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=gateway-nginx -o jsonpath='{.items[0].metadata.name}') kubectl logs $NGINX_POD_NAME -c nginx | grep coffee ``` The access log records the address of the machine you sent the request from. ## Troubleshooting ### No IngressLink is created Confirm the `--external-load-balancer` flag is set on the control plane deployment. Helm ignores values a chart does not define, so a chart without external load balancer support renders a deployment without the flag: ```shell kubectl get deploy -n nginx-gateway ngf-nginx-gateway-fabric \ -o jsonpath='{.spec.template.spec.containers[?(@.name=="nginx-gateway")].args}' ``` ### The IngressLink has no status F5 Container Ingress Services writes this status, so an empty status means it has not processed the resource. Wait up to two minutes for reconciliation, then check its logs: ```shell kubectl logs -n kube-system deploy/f5-cis-f5-bigip-ctlr ``` ### No address is allocated Confirm F5 Container Ingress Services was deployed with `args.ipam=true`, then check the F5 IPAM Controller logs: ```shell kubectl logs -n kube-system -l app=f5-ipam-controller --tail=20 ``` A label that does not match a configured pool is reported directly: ```text [PROV] IPAM LABEL: gatewaylink Not Found ``` Set `ipamLabel` on the `ExternalLoadBalancer` to a pool name from the `args.ip_range` map used when installing the F5 IPAM Controller. ### The AS3 declaration is rejected Read the BIG-IP response in the F5 Container Ingress Services logs, which usually names the problem: ```shell kubectl logs -n kube-system deploy/f5-cis-f5-bigip-ctlr | grep -E "AS3\]\[POST\]|response:" ``` ### F5 Container Ingress Services reports that AS3 is not installed The Pod is in `CrashLoopBackOff` and its logs contain `[ERROR] AS3 RPM is not installed on BIGIP`. F5 Container Ingress Services infers this from a 404 on the AS3 endpoint, so it also appears when AS3 is installed but not serving. See [Troubleshooting](https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/troubleshooting.html) in the F5 documentation. After restoring AS3, delete the Pod so it retries without waiting out its backoff: ```shell kubectl delete pod -n kube-system -l app=f5-cis-f5-bigip-ctlr ``` ### A pool is empty Confirm the type of the Gateway's Service matches the F5 Container Ingress Services `pool_member_type`, and that the Gateway has a listener on the port the pool was built for. A missing or invalid `certificateRefs` Secret leaves an HTTPS listener unprogrammed, so the Service never exposes port 443: ```shell kubectl get svc gateway-nginx -o jsonpath='{.spec.type}{"\n"}{.spec.ports}' kubectl describe gateways.gateway.networking.k8s.io gateway ``` ### NGINX logs show an internal address as the client The client address travels inside the PROXY protocol header. NGINX reads it only when both the connection address and the address inside the header are trusted, so an internal address in the log means the header never arrived or was discarded. Confirm the iRule is attached. Creating an iRule on BIG-IP does not attach it to anything: ```shell curl -sku "$BIGIP_USERNAME:$BIGIP_PASSWORD" "https://$BIGIP_ADDRESS/mgmt/tm/ltm/virtual" \ | python3 -c 'import sys,json for v in json.load(sys.stdin)["items"]: print(v["fullPath"], "->", v.get("rules", "no rules"))' ``` If the iRule is attached, confirm the trusted addresses: ```shell kubectl exec $NGINX_POD_NAME -c nginx -- grep set_real_ip_from /etc/nginx/conf.d/http.conf ``` Set `trustedAddresses` on the `NginxProxy` resource to the subnet of the IP address which the BIG-IP system uses to send traffic to NGINX. ### A configured field has no effect Kubernetes discards fields that are not in the installed custom resource definition schema without reporting an error, so both controllers report success while the field never arrives. Check where the field stops: ```shell export FIELD_NAME="ipamLabel" kubectl get crd ingresslinks.cis.f5.com -o yaml | grep -A5 "$FIELD_NAME" kubectl logs -n nginx-gateway deploy/ngf-nginx-gateway-fabric | grep "unknown field" kubectl get ingresslink gateway-nginx -o jsonpath='{.spec}' | python3 -m json.tool ``` An `unknown field` message means the installed custom resource definition is older than the NGINX Gateway Fabric release. Install a matching version. ## Remove the configuration Delete the `ExternalLoadBalancer` so F5 Container Ingress Services deletes the objects it created on BIG-IP: ```shell kubectl delete externalloadbalancer gateway-elb ``` Confirm the virtual servers are gone: ```shell curl -sku "$BIGIP_USERNAME:$BIGIP_PASSWORD" "https://$BIGIP_ADDRESS/mgmt/tm/ltm/virtual" | python3 -m json.tool | grep fullPath ``` ## References - [Distribute traffic across clusters with F5 BIG-IP](/ngf/external-loadbalancers/gateway-link/multicluster.md): terminate TLS at BIG-IP and spread traffic across two clusters, with health monitors and iRules. - [F5 IngressLink documentation](https://clouddocs.f5.com/containers/latest/userguide/ingresslink/): the F5 Container Ingress Services resource that NGINX Gateway Fabric generates. - [F5 Application Services 3 Extension reference](https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/refguide/schema-reference.html): the declaration format F5 Container Ingress Services posts to BIG-IP. - [NGINX Gateway Fabric](https://github.com/nginx/nginx-gateway-fabric): the NGINX Gateway Fabric source, including the `ExternalLoadBalancer` custom resource definitions. - [F5 Container Ingress Services](https://github.com/F5Networks/k8s-bigip-ctlr): the F5 Container Ingress Services source and custom resource definitions. - [F5 IPAM Controller](https://github.com/F5Networks/f5-ipam-controller): allocates virtual server addresses. - [F5 Container Ingress Services configuration parameters](https://clouddocs.f5.com/containers/latest/userguide/config-parameters.html): the full list of deployment options. - [PROXY protocol specification](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt): the header format the iRule generates. -------------------------------------------------------------------------------- title: "Get started with F5 WAF for NGINX (PLM)" url: https://docs.nginx.com/nginx-gateway-fabric/waf-integration/get-started-plm/index.md -------------------------------------------------------------------------------- Use this tutorial to set up end-to-end traffic protection with F5 WAF for NGINX using Policy Lifecycle Management (PLM). By the end, you'll have: - Deployed the PLM infrastructure (Policy Controller and SeaweedFS storage) - Connected NGINX Gateway Fabric to PLM storage - Defined a WAF policy using `APPolicy` and `APLogConf` custom resources - Attached a `WAFPolicy` to a Gateway and configured HTTPRoutes - Validated policy compilation and verified that attacks are blocked PLM is one of four WAF policy source types. With PLM, you define your security posture as `APPolicy` and `APLogConf` custom resources instead of compiling and hosting bundles yourself. For a comparison with the other source types, see [PLM (Policy Lifecycle Management)](/ngf/waf-integration/overview.md#plm-policy-lifecycle-management). ## Before you begin Before you start, make sure you have: - `kubectl` access to a Kubernetes cluster. - A valid F5 WAF for NGINX subscription. F5 WAF for NGINX is a separate add-on to NGINX Plus and isn't included with the NGINX Plus license. - Your F5 WAF for NGINX JWT from MyF5. To get it: 1. Log in to [MyF5](https://my.f5.com/manage/s/). 2. Go to **My Products & Plans** > **Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. Use a regular NGINX Plus instance JWT. - Your private registry credentials Secret for `private-registry.nginx.com`. You'll reference this Secret when you install NGINX Gateway Fabric. ### PLM prerequisites The following requirements apply to the PLM backend you'll install in this tutorial: - A Kubernetes cluster with a default StorageClass that supports dynamic provisioning. The PLM object store relies on PersistentVolumeClaims. Without a default StorageClass, the SeaweedFS pods stay `Pending`. - Helm 3.x installed. The PLM backend installs as a Helm chart. - An F5 WAF for NGINX JWT from MyF5, used to pull images from `private-registry.nginx.com`. - Optionally, `nginx-repo.crt` and `nginx-repo.key` from MyF5, needed only for authenticated signature updates from `pkgs.nginx.com`. PLM ships with an embedded SeaweedFS S3-compatible object store. The bundled SeaweedFS operator deploys and manages it. You don't need to provide an external S3 bucket. The PLM controller writes compiled policy bundles to this store; the data plane reads from it. By default, the deployment creates one master pod, one filer pod, and three volume pods, each backed by its own PVC. The chart generates credentials for the store and saves them in the `-f5-waf-seaweedfs-auth` Secret. By default, communication between PLM and the object store uses unencrypted HTTP. To enable TLS, see the PLM chart values (`helm show values nginx-stable/f5-waf-policy-controller`). ### Example values This tutorial uses the following example values. You can use different values — if you do, replace them consistently throughout. | Example value | What it represents | |---|---| | `plm-system` | Namespace for the PLM backend components | | `plm` | Helm release name for the PLM installation | | `` | F5 WAF for NGINX Policy Controller chart and image version | | `security` | Namespace for `APPolicy` and `APLogConf` resources | | `default` | Namespace for the Gateway and `WAFPolicy` | | `cafe.example.com` | Example hostname for HTTPRoutes | ## Deploy PLM infrastructure The Policy Lifecycle Manager (PLM) backend runs as a Kubernetes operator. It watches WAF custom resources and compiles WAF policies into bundles. The Policy Controller delegates compilation to a separate compiler service over gRPC. The resulting bundles are stored in an embedded SeaweedFS S3-compatible object store. F5 WAF for NGINX is installed using a separate Helm chart from your NGINX data plane. The steps in this section install only the F5 WAF for NGINX PLM components and do not affect your existing NGINX installation. ### Create the registry pull secret Create a namespace for the PLM components, store your JWT in a Kubernetes Secret, then create the registry pull secret for the private F5 container registry. 1. Create the namespace and store your JWT. The following commands assume your JWT file is named `license.jwt`: ```shell kubectl create namespace plm-system kubectl create secret generic jwt-reg-secret \ --namespace plm-system \ --from-file=license.jwt ``` 2. Retrieve the JWT from the Secret and create the registry pull secret: ```shell JWT=$(kubectl get secret jwt-reg-secret \ --namespace plm-system \ -o jsonpath='{.data.license\.jwt}' | base64 -d) kubectl create secret docker-registry regcred \ --namespace plm-system \ --docker-server=private-registry.nginx.com \ --docker-username="$JWT" \ --docker-password=none \ --dry-run=client --output yaml | kubectl apply -f - ``` ### Install the Policy Controller Create a values file for the Helm installation. The `securityUpdatesRepo.cert` and `securityUpdatesRepo.key` fields are optional. They are only required if your signature repository needs certificate-based authentication. The Policy Controller starts successfully with these fields left empty. If your signature repository requires them, replace `` and `` with the base64-encoded contents of your `nginx-repo.crt` and `nginx-repo.key` files. To encode them, run: ```shell base64 --wrap=0 < nginx-repo.crt base64 --wrap=0 < nginx-repo.key ``` Create `/tmp/plm-values.yaml`: ```yaml imagePullSecrets: - name: regcred securityUpdatesRepo: cert: "" # optional: only needed for authenticated signature repository access key: "" # optional: only needed for authenticated signature repository access policyController: image: tag: "" compiler: image: tag: "" seaweedfsOperatorConfig: seaweedfs: image: tag: "" seaweedfs-operator: image: tag: "" pullSecrets: regcred ``` #### Enable TLS for PLM storage (optional) By default, communication between PLM components and the SeaweedFS object store uses unencrypted HTTP. To enable TLS, add a `certificates` block to `/tmp/plm-values.yaml`: ```yaml seaweedfsOperatorConfig: seaweedfs: certificates: enabled: true ``` **Note:** The PLM chart does not generate certificates. You must create the five Secrets listed in the commands below before running `helm upgrade --install`. If any Secret is missing, the SeaweedFS pods will fail to mount their certificates and will not start. **Note:** If you're enabling TLS on an existing installation, the storage backend restarts and objects written before the switch can become orphaned. See the [APPolicy shows `invalid` with `unexpected EOF` after enabling TLS](#troubleshoot-the-deployment) entry in the troubleshooting section. A fresh installation with TLS enabled from the start doesn't have this issue. Create the Secrets from your CA and certificate files before installing. The chart expects Secret names in the form `-f5-waf-seaweedfs-` — for the `plm` release name used in this tutorial, those are: ```shell kubectl create secret generic plm-f5-waf-seaweedfs-ca-cert \ --namespace plm-system \ --from-file=tls.crt= \ --from-file=ca.crt= kubectl create secret tls plm-f5-waf-seaweedfs-master-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-volume-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-filer-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-client-cert \ --namespace plm-system \ --cert= \ --key= ``` The CA Secret requires both `tls.crt` and `ca.crt` keys, both pointing to the same CA certificate file. The PLM chart mounts the CA using `tls.crt` into the Policy Controller, compiler, and SeaweedFS pods. The data plane's S3 client reads `ca.crt` from the same Secret when verifying the storage endpoint. The four component Secrets use `kubectl create secret tls`, which produces `tls.crt` and `tls.key` — no `ca.crt` key is needed for them. Replace each `` placeholder with the path to the corresponding certificate and key file from your PKI. The CA must sign all component certificates. If you don't have an existing PKI, generate a CA and sign the five component certificates before proceeding. #### Install the chart Add the NGINX Helm repository and install the chart: ```shell helm repo add nginx-stable https://helm.nginx.com/stable helm repo update nginx-stable helm upgrade --install plm nginx-stable/f5-waf-policy-controller \ --version \ --namespace plm-system \ --values /tmp/plm-values.yaml ``` To see all available configuration options for the PLM chart, run: ```shell helm show values nginx-stable/f5-waf-policy-controller --version ``` ### Verify the deployment Wait for all PLM components to become ready. The Policy Controller's init container waits for both the compiler service and the SeaweedFS S3 endpoint to be available before it starts, so the controller pod will show `Init:0/1` until SeaweedFS is ready. Wait for the SeaweedFS storage backend: ```shell kubectl rollout status deployment/plm-seaweedfs-operator \ --namespace plm-system --timeout=120s ``` The SeaweedFS operator creates the SeaweedFS pods after it reconciles the SeaweedFS custom resource, so there is a window where the operator deployment is ready but no SeaweedFS pods exist yet. Poll until the pods appear and are ready: ```shell end=$((SECONDS + 300)) until kubectl wait pods \ --selector app.kubernetes.io/name=seaweedfs \ --for=condition=Ready \ --namespace plm-system \ --timeout=10s 2>/dev/null; do if [ $SECONDS -ge $end ]; then echo "Timed out waiting for SeaweedFS pods" exit 1 fi sleep 5 done ``` Wait for the Policy Controller: ```shell kubectl rollout status deployment/plm-f5-waf-policy-controller \ --namespace plm-system --timeout=180s ``` Confirm all pods are running: ```shell kubectl get pods --namespace plm-system ``` Example output: ```text NAME READY STATUS RESTARTS plm-f5-waf-compiler-service-xxxxx 1/1 Running 0 plm-f5-waf-policy-controller-xxxxx 1/1 Running 0 plm-seaweedfs-operator-xxxxx 1/1 Running 0 plm-f5-waf-seaweed-master-0 1/1 Running 0 plm-f5-waf-seaweed-filer-0 1/1 Running 0 plm-f5-waf-seaweed-volume-0 1/1 Running 0 plm-f5-waf-seaweed-volume-1 1/1 Running 0 plm-f5-waf-seaweed-volume-2 1/1 Running 0 ``` Confirm the CRDs are present: ```shell kubectl get crd | grep appprotect.f5.com ``` Expected output: ```text aplogconfs.appprotect.f5.com appolicies.appprotect.f5.com apsignatures.appprotect.f5.com apusersigs.appprotect.f5.com ``` All eight pods running and all four CRDs present confirms the PLM backend is ready. ### Update the CRDs **Note:** Skip this step on a fresh install — Helm installs the CRDs automatically. Only follow these steps when upgrading an existing PLM installation. When upgrading PLM, apply the CRDs manually before running `helm upgrade`: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/waf-policy-controller//manifests/1-deploy-crds.yaml ``` ### Troubleshoot the deployment These are the most common failures during PLM installation, roughly in order of likelihood. #### Pods stuck in `ImagePullBackOff` The JWT is wrong, expired, or contains a line break. Check the events log: ```shell kubectl get events --namespace plm-system --field-selector reason=Failed ``` Use the full JWT string as the registry username. Use the literal string `none` as the password. #### Policy Controller stuck in `Init:0/1` The `Init:0/1` state is expected during startup. The init container waits for the compiler service and the S3 endpoint before it starts. If the pod stays in `Init:0/1` for more than a few minutes, check that the SeaweedFS pods are `Running`: ```shell kubectl get pods --namespace plm-system --selector app.kubernetes.io/name=seaweedfs ``` The most common cause is PVCs stuck in `Pending` because the cluster has no default StorageClass. #### SeaweedFS pods `Pending` SeaweedFS pods stay `Pending` when the cluster has no default StorageClass or insufficient capacity. Check the PVCs and available storage classes: ```shell kubectl get pvc --namespace plm-system kubectl get storageclass ``` #### `APPolicy` shows `invalid` with `unexpected EOF` after enabling TLS Enabling TLS on an existing installation restarts the storage backend. Objects written before TLS was enabled can become orphaned. Check the filer log: ```shell kubectl logs --namespace plm-system plm-f5-waf-seaweed-filer-0 | grep "not found" ``` If the output contains `volume N not found`, orphaned objects exist. Delete the affected `APPolicy` resource and reapply it. The Policy Controller regenerates the bundle. #### Helm install fails on a ClusterRole If the error references `seaweed-editor-role` or `seaweed-viewer-role`, another PLM installation already exists in the cluster. Only one PLM installation is supported per cluster. Remove the existing release before installing. #### Check the Policy Controller logs Use the Policy Controller logs to diagnose any policy-related failure: ```shell kubectl logs --namespace plm-system deploy/plm-f5-waf-policy-controller -c policy-controller ``` **Note:** The `-c policy-controller` flag is required because the pod has more than one container. The containers are distroless, so `kubectl exec` isn't available for interactive debugging. ## Connect NGINX Gateway Fabric to PLM storage NGINX Gateway Fabric fetches compiled bundles from in-cluster PLM storage. You set up storage access once, cluster-wide, at install time. This configuration applies to every `WAFPolicy` that uses `type: PLM`. Create a `values.yaml` file that enables WAF and sets the PLM storage connection details under `nginxGateway.plmStorage`. #### HTTPS (secure) ```yaml nginxGateway: plmStorage: url: "https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local" credentialsSecretName: "plm-system/plm-f5-waf-seaweedfs-auth" # contains the seaweedfs_admin_secret field tls: caSecretName: "plm-ca-secret" # Secret with ca.crt for verifying the storage service clientSSLSecretName: "plm-client-secret" # Secret with tls.crt/tls.key for mutual TLS insecureSkipVerify: false # use only for testing ``` **Note:** Always use HTTPS with TLS verification (`caSecretName`) in production. Add `clientSSLSecretName` for mutual TLS in high-security environments, and never set `insecureSkipVerify: true`. **Note:** `credentialsSecretName` and `caSecretName` must reference Secrets in the NGINX Gateway Fabric control plane namespace, unless you prefix them with `/`. #### HTTP ```yaml nginxGateway: plmStorage: url: "http://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:8333" credentialsSecretName: "plm-system/plm-f5-waf-seaweedfs-auth" # contains the seaweedfs_admin_secret field ``` Install NGINX Gateway Fabric by following [the installation guide](/ngf/install/helm.md) and using the **NGINX Plus with WAF** tab. Apply this `values.yaml` file in your install or upgrade command by specifying `--values values.yaml`. The PLM installation creates the credentials Secret automatically, containing the S3 secret access key in the `seaweedfs_admin_secret` field (access key ID `admin` by default): ```yaml apiVersion: v1 kind: Secret metadata: name: plm-storage-credentials namespace: nginx-gateway type: Opaque data: seaweedfs_admin_secret: ``` NGINX Gateway Fabric reloads the PLM credentials and TLS Secrets when they change, so you can rotate credentials without restarting the pod. If you install NGINX Gateway Fabric using Kubernetes manifests, use the equivalent `plm-storage-*` flags documented in the [command-line reference](/ngf/reference/cli-help.md). ## Deploy the sample application Deploy the `customers` and `orders` sample applications. The `customers` app returns a response containing fake sensitive data (credit card number and SSN), which you'll use later to demonstrate data guard masking: ```yaml kubectl apply -f - < This section is typically owned by the security team. If you're not on the security team, share this section with them before continuing. Create the `security` namespace. The security team's `APPolicy` and `APLogConf` resources live here. In NGINX Gateway Fabric, the `ReferenceGrant` that permits cross-namespace WAFPolicy references also lives in this namespace. ```shell kubectl create namespace security ``` ## Configure security logging (optional) **Note:** Omit the `securityLogs` field when you create the `WAFPolicy` in [Deploy the Gateway and attach WAFPolicy](#deploy-the-gateway-and-attach-wafpolicy). This section is typically owned by the security team. If you're not on the security team, share this section with them before continuing. PLM security logging profiles are defined as `APLogConf` custom resources. Define a log profile that logs illegal requests: ```yaml kubectl apply -f - < The security team usually owns this section. They define the policy in the `security` namespace, separate from the Gateway namespace, so they can manage security resources independently from routing configuration. If you're not on the security team, share this section with them. You'll need the `APPolicy` name and namespace before continuing. The `APPolicy` resource defines the security policy. The PLM controller watches the resource, compiles the policy, and writes `status.bundle` with `state: ready` when the bundle is available. #### Inline Create an `APPolicy` resource with an inline policy that blocks all attack signatures: ```yaml kubectl apply -f - <`, ``, ``, ``, ``, and `` with your values: ```shell kubectl apply -f - <<'EOF' apiVersion: appprotect.f5.com/v1 kind: APPolicy metadata: name: namespace: spec: policy: $ref: externalReferenceDetails: repositoryDetails: repository: https://github.com//.git ref: "" EOF ``` **Note:** Pin `ref` to a tag or commit SHA rather than a branch name in production environments. Check that the bundle compiled successfully: ```shell kubectl get appolicy \ --namespace \ --output jsonpath='State: {.status.bundle.state}{"\n"}Bundle: {.status.bundle.location}{"\n"}Compiler: {.status.bundle.compilerVersion}{"\n"}' ``` The output shows `State: ready` when compilation succeeds. #### Private repository For private repositories, create a Kubernetes Secret with your personal access token (PAT): ```shell kubectl create secret generic git-token-secret \ --namespace \ --from-literal=token= ``` Then reference the Secret in the `APPolicy` resource: ```shell kubectl apply -f - <<'EOF' apiVersion: appprotect.f5.com/v1 kind: APPolicy metadata: name: namespace: spec: policy: $ref: externalReferenceDetails: repositoryDetails: repository: https://github.com//.git ref: "" authentication: token: git-token-secret EOF ``` #### Update a Git-referenced policy The Policy Controller doesn't poll the Git repository for changes. It fetches the policy file when the `APPolicy` spec changes. To pick up a new version of the policy, push your changes to the repository. Then update `ref` in the `APPolicy` resource to the new tag or commit SHA and reapply the resource. Reapplying an unchanged `APPolicy` doesn't trigger a fetch. Changing an annotation doesn't trigger a fetch either. If you need to re-fetch the same `ref` (for example, after force-updating a tag), delete the `APPolicy` resource and recreate it. #### Precompiled bundle The precompiled-bundle method lets you reference a `.tgz` policy bundle stored in an artifact registry (for example, Artifactory or Nexus). The Policy Controller imports the bundle and stores it in the SeaweedFS object store without recompiling the bundle. Use this method when: - Your security team compiles and publishes bundles through an external pipeline. - You want to separate policy compilation from cluster operations. Create an `APPolicy` resource that references your bundle. Replace ``, ``, and `` with your values: ```shell kubectl apply -f - <<'EOF' apiVersion: appprotect.f5.com/v1 kind: APPolicy metadata: name: namespace: security spec: policy: $ref: "https:///.tgz" EOF ``` **Note:** The Policy Controller must reach the artifact registry over HTTPS. If the registry uses a private certificate authority (CA), mount the CA certificate into the Policy Controller pod and set the `SSL_CERT_FILE` environment variable to its path. `SSL_CERT_FILE` replaces the system trust store entirely. It doesn't append to the system trust store. If SeaweedFS TLS is also turned on, combine both CAs into a single file and reference that file. If the `APPolicy` status shows `x509: certificate signed by unknown authority`, the Policy Controller doesn't trust the artifact registry CA. Check the status for the full error: ```shell kubectl describe appolicy --namespace security ``` #### Confirm the policy is ready The Policy Controller processes the bundle and updates the `APPolicy` status. Check the `bundle.state` field: ```shell kubectl get appolicy \ --namespace security \ --output jsonpath='State: {.status.bundle.state}{"\n"}Bundle: {.status.bundle.location}{"\n"}isCompiled: {.status.processing.isCompiled}{"\n"}' ``` When the bundle is ready, the output looks like this: ```text State: ready Bundle: s3://plm-system/bundles/_imported_.tgz isCompiled: false ``` `isCompiled: false` confirms the bundle was imported without recompilation. `bundle.state` can be one of: | State | Meaning | |-------|---------| | `pending` | The Policy Controller hasn't yet processed the resource. | | `processing` | The Policy Controller is importing or storing the bundle. | | `ready` | The bundle is stored and ready to use. The Policy Controller has populated `bundle.location`. | | `invalid` | The Policy Controller couldn't import the bundle. Check the status for error detail. | #### Update a precompiled bundle The Policy Controller doesn't poll the artifact registry for changes. To pick up a new version of a bundle, update the `$ref` URL in your `APPolicy` resource and reapply the resource. Changing an annotation doesn't trigger a new download. If you need to re-fetch the same URL, delete the `APPolicy` resource and recreate it. Replace ``, ``, and `` with your values: ```shell kubectl apply -f - <<'EOF' apiVersion: appprotect.f5.com/v1 kind: APPolicy metadata: name: namespace: security spec: policy: $ref: "https:///.tgz" EOF ``` The `APPolicy` and `APLogConf` are in the `security` namespace, but the `WAFPolicy` you create next targets a Gateway in the `default` namespace. To permit the cross-namespace reference, create a `ReferenceGrant` in the `security` namespace: ```yaml kubectl apply -f - < ``` **Verify normal traffic flows.** Send a request to the `customers` route — the response contains the fake sensitive data from the `customers` backend: **Note:** If you have a DNS record for `cafe.example.com`, you can send the request directly to that hostname without `--resolve`. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/customers ``` ```text Customer List: Name: John Doe Credit Card: 4111-1111-1111-1111 SSN: 123-45-6789 ``` The sensitive data passes through because the gateway-level `attack-signatures` policy only inspects inbound requests for attack patterns — it doesn't mask outbound response data. **Verify attacks are blocked.** Send a request with a cross-site scripting (XSS) payload: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP "http://cafe.example.com:$GW_PORT/customers?x=" ``` The WAF detects the attack signature and rejects the request: ```text Request Rejected ... ``` **Verify the `orders` route is also protected.** Because the policy targets the Gateway, all attached routes inherit protection: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP "http://cafe.example.com:$GW_PORT/orders?x=" ``` ```text Request Rejected ... ``` **Note:** The exact blocking response depends on your WAF policy configuration. Check the security log for a corresponding blocked event using `kubectl logs -c waf-enforcer`. ## Apply a route-level override (optional) The `customers` route returns sensitive data (credit card numbers and SSNs) in the response body. The gateway-level policy blocks inbound attacks but doesn't inspect outbound responses. This is a common pattern for SecOps and app team collaboration: the security team defines a stricter policy for a specific service, and the platform engineer or app developer attaches it as a route-level override. The override applies only to the `customers` route — other routes continue using the gateway-level policy. **Note:** Only one `WAFPolicy` can target a given resource at a given level. If a second `WAFPolicy` targets the same Gateway or route, it is rejected with `Accepted=False` and reason `Conflicted`. See [Policy attachment](/ngf/waf-integration/overview.md#policy-attachment). To protect sensitive data in responses, define a **data guard** `APPolicy` and apply it as a route-level override on the `customers` route: ```yaml kubectl apply -f - <for
cafe.example.com"] subgraph cluster [Kubernetes Cluster] subgraph appNs [Namespace
default] subgraph nsPadding [" "] nginxPod[Pod
NGINX] coffeeV1Pod[Pod
coffee v1] coffeeV2Pod[Pod
coffee v2] teaPod[Pod
tea] teaPostPod[Pod
tea-post] end end end ngfSvc --> nginxPod nginxPod --/coffee--> coffeeV1Pod nginxPod --/coffee
header: version=v2
OR
/coffee?TEST=v2--> coffeeV2Pod nginxPod --GET /tea--> teaPod nginxPod --POST /tea--> teaPostPod users --> ngfSvc class clusterPadding,nsPadding,clusterPadding2 noBorder class gwNS,appNs namespace class ngfSvc,nginxPod nginxNode class coffeeV1Pod,coffeeV2Pod coffeeNode class teaPod,teaPostPod teaNode classDef noBorder stroke:none,fill:none classDef default fill:#FFFFFF,stroke:#000000 classDef namespace fill:#FFFFFF,stroke:#036ffc,stroke-dasharray: 5 5,text-align:center classDef nginxNode fill:#b4e0ad,stroke:#2AA317 classDef coffeeNode fill:#edbd8c,stroke:#D9822B classDef teaNode fill:#ff8f6a,stroke:#e5805f ``` The goal is to create a set of rules that will result in client requests being sent to specific backends based on the request attributes. In this diagram, we have two versions of the `coffee` service. Traffic for v1 needs to be directed to the old application, while traffic for v2 needs to be directed towards the new application. We also have two `tea` services, one that handles GET operations and one that handles POST operations. Both the `tea` and `coffee` applications share the same Gateway. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Coffee applications ### Deploy the Coffee applications Begin by deploying the `coffee-v1`, `coffee-v2` and `coffee-v3` applications: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/advanced-routing/coffee.yaml ``` ### Deploy the Gateway API Resources for the Coffee applications The [gateway](https://gateway-api.sigs.k8s.io/api-types/gateway/) resource is typically deployed by the [cluster operator](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#roles-and-personas_1). To deploy the gateway: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. The [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) is typically deployed by the [application developer](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#roles-and-personas_1). To deploy the `coffee` HTTPRoute: ```yaml kubectl apply -f - < remote-kubeconfig.yaml </` form. F5 Container Ingress Services reads the list of external clusters and their kubeconfig Secrets from it, so without this value it has no way to reach cluster B. - `args.pool_member_type` must match the type of the Gateway's Service. Use `nodeport` with `NodePort`, or `cluster` with `ClusterIP`. - `args.log-as3-response=true` logs the BIG-IP response to each declaration, which is useful for troubleshooting. Confirm F5 Container Ingress Services reached BIG-IP and accepted the mode: ```shell kubectl logs -n kube-system deploy/f5-cis-f5-bigip-ctlr | grep -E "authn/login|multi-cluster-mode" ``` The log shows a successful login and the configured multi-cluster mode: ```text [DEBUG] [BIGIP] postConfig request: POST https://192.0.2.10:443/mgmt/shared/authn/login 200 OK [DEBUG] Multi-cluster-mode: standalone, local cluster name: local ``` ## Set up for both clusters Apply the following resources to **both** clusters. Use the same Gateway name and the same listeners in each, so the data plane Services carry matching labels and expose the same ports. F5 Container Ingress Services builds one virtual server per Service port and pools every cluster behind that virtual server. ### Install the custom resource definitions Install the F5 Container Ingress Services custom resource definitions in **both** clusters, including cluster B, which does not run F5 Container Ingress Services: ```shell kubectl apply -f https://raw.githubusercontent.com/F5Networks/k8s-bigip-ctlr/v/docs/config_examples/customResourceDefinitions/customresourcedefinitions.yml ``` Confirm the `IngressLink` custom resource definition is installed: ```shell kubectl get crd ingresslinks.cis.f5.com ``` ```text NAME CREATED AT ingresslinks.cis.f5.com 2026-08-05T01:40:54Z ``` With external load balancer support enabled, NGINX Gateway Fabric watches `IngressLink` resources on startup in every cluster it runs in. A cluster without the custom resource definition leaves the control plane unable to start, and its Pod restarts continuously: ```text no matches for kind "IngressLink" in version "cis.f5.com/v1" failed to start control loop: failed to wait for provisioner-IngressLink caches to sync ``` ### Install NGINX Gateway Fabric [Install](/ngf/install/) NGINX Gateway Fabric with external load balancer support enabled. Using Helm, set the `nginxGateway.externalLoadBalancer.enable=true` value. Using Kubernetes manifests, add the `--external-load-balancer` flag to the `nginx-gateway` container arguments. ### Create a Gateway Create an `NginxProxy` resource named `gatewaylink-proxy`, which exposes the readiness probe: ```yaml kubectl apply -f - <", v.get("rules", "no rules"))' ``` If the iRule is attached, confirm the trusted addresses: ```shell kubectl exec $NGINX_POD_NAME -c nginx -- grep set_real_ip_from /etc/nginx/conf.d/http.conf ``` Set `trustedAddresses` on the `NginxProxy` resource to the subnet of the IP address which the BIG-IP system uses to send traffic to NGINX. ### A configured field has no effect Kubernetes discards fields that are not in the installed custom resource definition schema without reporting an error, so both controllers report success while the field never arrives. Check where the field stops: ```shell export FIELD_NAME="ipamLabel" kubectl get crd ingresslinks.cis.f5.com -o yaml | grep -A5 "$FIELD_NAME" kubectl logs -n nginx-gateway deploy/ngf-nginx-gateway-fabric | grep "unknown field" kubectl get ingresslink gateway-nginx -o jsonpath='{.spec}' | python3 -m json.tool ``` An `unknown field` message means the installed custom resource definition is older than the NGINX Gateway Fabric release. Install a matching version. ### The control plane restarts continuously in cluster B The NGINX Gateway Fabric Pod reports `CrashLoopBackOff`, and its logs end with a cache sync failure: ```text no matches for kind "IngressLink" in version "cis.f5.com/v1" failed to start control loop: failed to wait for provisioner-IngressLink caches to sync ``` The F5 Container Ingress Services custom resource definitions are missing from that cluster. With external load balancer support enabled, NGINX Gateway Fabric watches `IngressLink` resources on startup, whether or not Container Ingress Services runs there. - Install the custom resource definitions in the affected cluster: ```shell kubectl apply -f https://raw.githubusercontent.com/F5Networks/k8s-bigip-ctlr/v/docs/config_examples/customResourceDefinitions/customresourcedefinitions.yml ``` - Delete the Pod so it restarts immediately rather than waiting out its backoff: ```shell kubectl delete pod -n nginx-gateway -l app.kubernetes.io/name=nginx-gateway-fabric ``` ### The remote cluster has no pool Only `_local` pools exist on BIG-IP, and traffic never reaches cluster B. F5 Container Ingress Services could not load the cluster B kubeconfig, so it has no endpoints to pool. Start with its log, which names the cause directly: ```shell kubectl logs -n kube-system deploy/f5-cis-f5-bigip-ctlr | grep -i "MultiCluster" ``` - Confirm the Secret exists under the name and namespace the `extended-spec-config` ConfigMap refers to. A Secret created under a different name is reported as missing: ```text error occurred while fetching Secret: remote-kubeconfig for the cluster: remote, Error: secrets "remote-kubeconfig" not found ``` - Confirm the token is still valid. A token issued for a ServiceAccount that has since been deleted and recreated is rejected: ```text the server has asked for the client to provide credentials ``` Regenerate the kubeconfig on cluster B and recreate the Secret. - Confirm the Secret holding the cluster B kubeconfig parses. A kubeconfig with broken indentation is stored without complaint and fails only when F5 Container Ingress Services loads it: ```shell kubectl get secret remote-kubeconfig -n kube-system -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/check.yaml KUBECONFIG=/tmp/check.yaml kubectl get nodes ``` The command lists the cluster B nodes. An error such as `mapping values are not allowed in this context` means the file is malformed, so regenerate it and recreate the Secret. - Confirm the `clusterName` in the extended spec ConfigMap matches the `clusterName` under `remoteClusters` in the `ExternalLoadBalancer`. - Restart F5 Container Ingress Services after replacing the Secret, because it reads the kubeconfig at startup: ```shell kubectl rollout restart deploy/f5-cis-f5-bigip-ctlr -n kube-system ``` ### The remote pool member is down The `_remote` pool exists but its member reports `offline`, so all traffic goes to the local cluster. - Confirm the cluster B data plane Service exposes the same ports as cluster A. A missing HTTPS listener leaves nothing listening on the 443 NodePort: ```shell kubectl get svc gateway-nginx -o jsonpath='{range .spec.ports[*]}{.name} {.port}:{.nodePort}{"\n"}{end}' ``` - Confirm the `nginx-tls` Secret exists in cluster B. Without it the HTTPS listener is not programmed and NGINX never listens on 443. - Restart the data plane after creating a certificate. NGINX does not load a certificate created after the Pod started, and the Gateway reports every condition as healthy while the listener is missing from the configuration: ```shell kubectl rollout restart deploy/gateway-nginx -n default ``` ### Requests fail with a connection reset A request through BIG-IP fails with `Recv failure: Connection reset by peer`. NGINX Gateway Fabric enables HTTP/2 by default. BIG-IP SSL profiles do not negotiate HTTP/2 unless configured to, so BIG-IP sends HTTP/1.1 into a connection NGINX set up for HTTP/2. - Set `disableHTTP2: true` on the `NginxProxy` resource, or use a BIG-IP SSL profile with HTTP/2 enabled. Confirm the setting reached the data plane rather than trusting the resource: ```shell kubectl exec $NGINX_POD_NAME -c nginx -- grep "listen 443" /etc/nginx/conf.d/http.conf ``` The absence of an `http2` token on the `listen` line means HTTP/2 is off. ## Remove the configuration Delete the `ExternalLoadBalancer` so F5 Container Ingress Services deletes the objects it created on BIG-IP: ```shell kubectl delete externalloadbalancer gateway-elb ``` Confirm the virtual servers are gone: ```shell curl -sku "$BIGIP_USERNAME:$BIGIP_PASSWORD" "https://$BIGIP_ADDRESS/mgmt/tm/ltm/virtual" | python3 -m json.tool | grep fullPath ``` ## References - [F5 IngressLink documentation](https://clouddocs.f5.com/containers/latest/userguide/ingresslink/): the F5 Container Ingress Services resource that NGINX Gateway Fabric generates. - [F5 Application Services 3 Extension reference](https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/refguide/schema-reference.html): the declaration format F5 Container Ingress Services posts to BIG-IP. - [F5 Container Ingress Services](https://github.com/F5Networks/k8s-bigip-ctlr): the F5 Container Ingress Services source and custom resource definitions. - [F5 IPAM Controller](https://github.com/F5Networks/f5-ipam-controller): allocates virtual server addresses when using the `ipamLabel` field instead of a fixed address. - [F5 BIG-IP iControl REST API](https://clouddocs.f5.com/api/icontrol-rest/): the API used by the `curl` commands in this guide. - [BIG-IP Virtual Edition on Amazon Web Services](https://clouddocs.f5.com/cloud/public/v1/aws_index.html) - [BIG-IP Virtual Edition on Microsoft Azure](https://clouddocs.f5.com/cloud/public/v1/azure_index.html) - [BIG-IP Virtual Edition on Google Cloud Platform](https://clouddocs.f5.com/cloud/public/v1/google_index.html) - [F5 Container Ingress Services multi-cluster guide](https://clouddocs.f5.com/containers/latest/userguide/multicluster/): multi-cluster deployment topologies. -------------------------------------------------------------------------------- title: "Gateway API compatibility" url: https://docs.nginx.com/nginx-gateway-fabric/overview/gateway-api-compatibility/index.md -------------------------------------------------------------------------------- This page describes which Gateway API resources NGINX Gateway Fabric supports and to which level. For information about NGINX Gateway Fabric and Gateway API version compatibility, view the [Technical specifications](/ngf/overview/technical-specifications.md) topic. ## Summary **Note:** You can view the [Milestone Roadmap](https://github.com/orgs/nginx/projects/10/views/5) in the NGINX Gateway Fabric GitHub project to see what API resources will be supported in upcoming releases. | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |---------------------------------------|---------------------|------------------------|---------------------------------------|-------------|---------------------| | [GatewayClass](/ngf/overview/gateway-api-compatibility.md#gatewayclass) | Supported | Not supported | Supported | v1 | Standard | | [Gateway](/ngf/overview/gateway-api-compatibility.md#gateway) | Supported | Partially supported | Not supported | v1 | Standard | | [HTTPRoute](/ngf/overview/gateway-api-compatibility.md#httproute) | Supported | Partially supported | Not supported | v1 | Standard | | [GRPCRoute](/ngf/overview/gateway-api-compatibility.md#grpcroute) | Supported | Partially supported | Not supported | v1 | Standard | | [ReferenceGrant](/ngf/overview/gateway-api-compatibility.md#referencegrant) | Supported | N/A | Not supported | v1 | Standard | | [TLSRoute](/ngf/overview/gateway-api-compatibility.md#tlsroute) | Supported | Not supported | Not supported | v1 | Standard | | [TCPRoute](/ngf/overview/gateway-api-compatibility.md#tcproute) | Supported | Supported | Not supported | v1 | Standard | | [UDPRoute](/ngf/overview/gateway-api-compatibility.md#udproute) | Supported | Supported | Not supported | v1 | Standard | | [BackendTLSPolicy](/ngf/overview/gateway-api-compatibility.md#backendtlspolicy) | Partially supported | Supported | Partially supported | v1 | Standard | | [Custom policies](/ngf/overview/gateway-api-compatibility.md#custom-policies) | N/A | N/A | Supported | N/A | N/A | ## Terminology Gateway API features has three [support levels](https://gateway-api.sigs.k8s.io/concepts/conformance/#2-support-levels): Core, Extended and Implementation-specific. We use the following terms to describe the support status for each level and resource field: - _Supported_. The resource or field is fully supported. - _Partially supported_. The resource or field is supported partially, with limitations. It will become fully supported in future releases. - _Not supported_. The resource or field is not yet supported. It will become partially or fully supported in future releases. **Note:** It's possible that NGINX Gateway Fabric will never support some resources or fields of the Gateway API. They will be documented on a case by case basis. Please note that while we make every effort to reflect the support status of experimental fields in our code and documentation, there may be instances where this is not explicitly indicated. Support for such fields is provided on a best-effort basis. ## Resources Each resource below includes the support status of their corresponding fields. For a description of each field, visit the [Gateway API documentation](https://gateway-api.sigs.k8s.io/references/spec/). ### GatewayClass | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |--------------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | GatewayClass | Supported | Not supported | Supported | v1 | Standard | NGINX Gateway Fabric supports a single GatewayClass resource configured with the `--gatewayclass` flag of the [controller](/ngf/reference/cli-help.md#controller) command. **Fields**: - `spec` - `controllerName` - supported. - `parametersRef` - NginxProxy resource supported. - `description` - supported. - `status` - `conditions` - supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/True/InvalidParameters` - `Accepted/False/UnsupportedVersion` - `Accepted/False/GatewayClassConflict` - `SupportedVersion/True/SupportedVersion` - `SupportedVersion/False/UnsupportedVersion` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/ParametersRefNotFound` - `ResolvedRefs/False/ParametersRefInvalid` - `supportedFeatures` - supported. ### Gateway | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |----------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | Gateway | Supported | Partially supported | Not supported | v1 | Standard | NGINX Gateway Fabric supports multiple Gateway resources. The Gateway resources must reference NGINX Gateway Fabric's corresponding GatewayClass. See the [controller](/ngf/reference/cli-help.md#controller) command for more information. **Fields**: - `spec` - `gatewayClassName`: Supported. - `infrastructure`: Supported. - `parametersRef`: NginxProxy resource supported. - `labels`: Supported. - `annotations`: Supported. - `listeners` - `name`: Supported. - `hostname`: Supported. - `port`: Supported. - `protocol`: Supported. - `tls` - `mode`: Supported. - `certificateRefs` - The TLS certificate and key must be stored in a Secret resource of type `kubernetes.io/tls`. - `options`: The options `nginx.org/ssl-protocols`, `nginx.org/ssl-ciphers`, `nginx.org/ssl-prefer-server-ciphers`, `nginx.org/ssl-session-cache`, `nginx.org/ssl-session-timeout` and `nginx.org/ssl-ecdh-curve` are supported. See [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) for more information. - `allowedRoutes`: Supported. - `addresses`: Valid IPAddresses will be added to the `externalIP` field in the related Services fronting NGINX. Users should ensure that the IP Family of the address matches the IP Family set in the NginxProxy resource (default is dual, meaning both IPv4 and IPv6), otherwise there may be networking issues. - `type`: Partially supported. Allowed value: `IPAddress`. - `value`: Partially supported. Dynamic address allocation when value is unspecified is not supported. - `TLS`: - `frontend`: - `default`: - `caCertificateRefs`: Supported. - `perPort`: - `caCertificateRefs`: Supported. - `backend`: - `clientCertificateRef`: Supported. - `allowedListeners`: Supported. - `defaultScope`: Not supported. - `status` - `addresses`: Partially supported (LoadBalancer and ClusterIP). - `attachedListenerSets`: Supported. - `conditions`: Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/True/ListenersNotValid` - `Accepted/True/InvalidParameters` - `Accepted/True/UnsupportedField` - `Accepted/False/ListenersNotValid` - `Accepted/False/Invalid` - `Accepted/False/UnsupportedValue` - `Accepted/False/UnsupportedAddress` - `Programmed/True/Programmed` - `Programmed/False/Invalid` - `Programmed/False/UnsupportedValue` - `Programmed/False/AddressNotUsable` - `Programmed/False/AddressNotAssigned` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/ParametersRefNotFound` - `ResolvedRefs/False/ParametersRefInvalid` - `ResolvedRefs/False/InvalidClientCertificateRef` - `ResolvedRefs/False/RefNotPermitted` - `InsecureFrontendValidationMode/True/ConfigurationChanged` - `listeners` - `name`: Supported. - `supportedKinds`: Supported. - `attachedRoutes`: Supported. - `conditions`: Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/False/UnsupportedProtocol` - `Accepted/False/InvalidCertificateRef` - `Accepted/False/NoValidCACertificate` - `Accepted/False/ProtocolConflict` - `Accepted/False/HostnameConflict` - `Accepted/False/UnsupportedValue` - `Accepted/False/RefNotPermitted` - `Programmed/True/Programmed` - `Programmed/False/Invalid` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/InvalidCertificateRef` - `ResolvedRefs/False/InvalidCACertificateRef` - `ResolvedRefs/False/InvalidRouteKinds` - `ResolvedRefs/False/RefNotPermitted` - `Conflicted/True/ProtocolConflict` - `Conflicted/True/HostnameConflict` - `Conflicted/False/NoConflicts` - `OverlappingTLSConfig/True/OverlappingHostnames` ### HTTPRoute | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |-----------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | HTTPRoute | Supported | Partially supported | Not supported | v1 | Standard | **Fields**: - `spec` - `parentRefs`: Supported. - `hostnames`: Supported. - `rules` - `matches` - `path`: Partially supported. Only `PathPrefix` and `Exact` types. - `headers`: Supported. - `queryParams`: Supported. - `method`: Supported. - `filters` - `type`: Supported. - `requestRedirect`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. Incompatible with `urlRewrite`. - `requestHeaderModifier`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. - `urlRewrite`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. Incompatible with `requestRedirect`. - `responseHeaderModifier`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. - `requestMirror`: Supported. Multiple mirrors can be specified. Percent and fraction-based mirroring are supported. - `cors`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. - `extensionRef`: Supported for SnippetsFilters and AuthenticationFilters. - `externalAuth`: Supported. If multiple filters are configured, NGINX Gateway Fabric uses the first and ignores the rest. - `backendRefs`: Partially supported. Backend ref `filters` are not supported. - `name`: Not supported. - `timeouts`: Not supported. - `retry`: Not supported. - `sessionPersistence`: Supported (NGINX Plus). - `status` - `parents` - `parentRef`: Supported. - `controllerName`: Supported. - `conditions`: Partially supported. Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/True/UnsupportedField` - `Accepted/False/NoMatchingListenerHostname` - `Accepted/False/NoMatchingParent` - `Accepted/False/NotAllowedByListeners` - `Accepted/False/UnsupportedValue` - `Accepted/False/InvalidListener` - `Accepted/False/HostnameConflict` - `Accepted/False/MultipleRoutesOnListener` - `Accepted/False/InvalidGateway` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/InvalidKind` - `ResolvedRefs/False/RefNotPermitted` - `ResolvedRefs/False/BackendNotFound` - `ResolvedRefs/False/UnsupportedValue` - `ResolvedRefs/False/InvalidIPFamily` - `ResolvedRefs/False/UnsupportedProtocol` - `ResolvedRefs/False/InvalidFilter` - `ResolvedRefs/False/InvalidInferencePool` - `PartiallyInvalid/True/UnsupportedValue` **Note:** If `name`, `timeouts`, or `retry` are defined for a HTTPRoute rule, they will be ignored and rule still will be created. ### GRPCRoute | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |-----------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | GRPCRoute | Supported | Partially supported | Not supported | v1 | Standard | **Fields**: - `spec` - `parentRefs`: Supported. - `hostnames`: Supported. - `rules` - `matches` - `method`: Partially supported. Only `Exact` type with both `method.service` and `method.method` specified. - `headers`: Supported - `filters` - `type`: Supported. - `requestHeaderModifier`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. - `responseHeaderModifier`: Supported. If multiple filters are configured, NGINX Gateway Fabric will choose the first and ignore the rest. - `requestMirror`: Supported. Multiple mirrors can be specified. - `extensionRef`: Supported for SnippetsFilters. - `backendRefs`: Partially supported. Backend ref `filters` are not supported. - `name`: Not supported. - `sessionPersistence`: Supported (NGINX Plus). - `status` - `parents` - `parentRef`: Supported. - `controllerName`: Supported. - `conditions`: Partially supported. Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/True/UnsupportedField` - `Accepted/False/NoMatchingListenerHostname` - `Accepted/False/NoMatchingParent` - `Accepted/False/NotAllowedByListeners` - `Accepted/False/UnsupportedValue` - `Accepted/False/InvalidListener` - `Accepted/False/HostnameConflict` - `Accepted/False/MultipleRoutesOnListener` - `Accepted/False/InvalidGateway` - `Accepted/False/UnsupportedConfiguration` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/InvalidKind` - `ResolvedRefs/False/RefNotPermitted` - `ResolvedRefs/False/BackendNotFound` - `ResolvedRefs/False/UnsupportedValue` - `ResolvedRefs/False/InvalidIPFamily` - `ResolvedRefs/False/UnsupportedProtocol` - `ResolvedRefs/False/InvalidFilter` - `PartiallyInvalid/True/UnsupportedValue` **Note:** If `name` is defined for a GRPCRoute rule, it will be ignored and rule still will be created. ### ReferenceGrant | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |----------------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | ReferenceGrant | Supported | N/A | Not supported | v1 | Standard | Fields: - `spec` - `to` - `group` - supported. - `kind` - supported. - `name`- supported. - `from` - `group` - supported. - `kind` - supported. - `namespace`- supported. ### TLSRoute | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |----------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | TLSRoute | Supported | Not supported | Not supported | v1 | Standard | **Fields**: - `spec` - `parentRefs`: Supported. - `hostnames`: Supported. - `rules` - `backendRefs`: Partially supported. Only one backend ref allowed. - `weight`: Not supported. - `status` - `parents` - `parentRef`: Supported. - `controllerName`: Supported. - `conditions`: Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/False/NoMatchingListenerHostname` - `Accepted/False/NoMatchingParent` - `Accepted/False/NotAllowedByListeners` - `Accepted/False/UnsupportedValue` - `Accepted/False/InvalidListener` - `Accepted/False/HostnameConflict` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/InvalidKind` - `ResolvedRefs/False/RefNotPermitted` - `ResolvedRefs/False/BackendNotFound` - `ResolvedRefs/False/UnsupportedValue` - `PartiallyInvalid/True/UnsupportedValue` ### TCPRoute | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |----------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | TCPRoute | Supported | Supported | Not supported | v1 | Standard | ### UDPRoute | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |----------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | UDPRoute | Supported | Supported | Not supported | v1 | Standard | ### BackendTLSPolicy | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |------------------|---------------------|------------------------|---------------------------------------|-------------|---------------------| | BackendTLSPolicy | Supported | Supported | Partially Supported | v1 | Standard | Fields: - `spec` - `targetRefs` - `group`: Supported. - `kind`: Supports `Service`. - `name`: Supported. - `validation` - `caCertificateRefs`: Supports single reference to a `ConfigMap` or `Secret`, with the CA certificate in a key named `ca.crt`. - `name`: Supported. - `group`: Supported. - `kind`: Supports `ConfigMap` and `Secret`. - `hostname`: Supported. - `wellKnownCertificates`: Supports `System`. This will set the CA certificate to the Alpine system root CA path `/etc/ssl/cert.pem`. NB: This option will only work if the NGINX image used is Alpine based. The NGF NGINX images are Alpine based by default. - `subjectAltNames`: Not supported. - `options`: Not supported. - `status` - `ancestors` - `ancestorRef`: Supported. - `controllerName`: Supported. - `conditions`: Partially supported. Supported (Condition/Status/Reason): - `Accepted/True/PolicyReasonAccepted` - `Accepted/False/PolicyReasonInvalid` - `Accepted/False/NoValidCACertificate` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/InvalidCACertificateRef` - `ResolvedRefs/False/InvalidKind` **Note:** If multiple `backendRefs` are defined for a HTTPRoute rule, all the referenced Services *must* have matching BackendTLSPolicy configuration. BackendTLSPolicy configuration is considered to be matching if 1. CACertRefs reference the same ConfigMap, or 2. WellKnownCACerts are the same, and 3. Hostname is the same. ### ListenerSet | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |-------------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | ListenerSet | Supported | Supported | Not supported | v1 | Standard | **Fields**: - `spec` - `parentRef`: Supported. - `listeners` - `name`: Supported. - `hostname`: Supported. - `port`: Supported. - `protocol`: Supported. - `tls` - `mode`: Supported. - `certificateRefs` - The TLS certificate and key must be stored in a Secret resource of type `kubernetes.io/tls`. - `options`: The options `nginx.org/ssl-protocols`, `nginx.org/ssl-ciphers`, `nginx.org/ssl-prefer-server-ciphers`, `nginx.org/ssl-session-cache`, `nginx.org/ssl-session-timeout` and `nginx.org/ssl-ecdh-curve` are supported. See [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) for more information. - `allowedRoutes`: Supported. - `status` - `conditions`: Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/True/ListenersNotValid` - `Accepted/False/NotAllowed` - `Accepted/False/ParentNotAllowed` - `Accepted/False/ListenersNotValid` - `Programmed/True/Programmed` - `Programmed/False/NotAllowed` - `Programmed/False/ParentNotAllowed` - `Programmed/False/ListenersNotValid` - `listeners` - `name`: Supported. - `supportedKinds`: Supported. - `attachedRoutes`: Supported. - `conditions`: Supported (Condition/Status/Reason): - `Accepted/True/Accepted` - `Accepted/False/UnsupportedProtocol` - `Accepted/False/InvalidCertificateRef` - `Accepted/False/ProtocolConflict` - `Accepted/False/HostnameConflict` - `Accepted/False/UnsupportedValue` - `Accepted/False/RefNotPermitted` - `Programmed/True/Programmed` - `Programmed/False/Invalid` - `ResolvedRefs/True/ResolvedRefs` - `ResolvedRefs/False/InvalidCertificateRef` - `ResolvedRefs/False/InvalidRouteKinds` - `ResolvedRefs/False/RefNotPermitted` - `Conflicted/True/ProtocolConflict` - `Conflicted/True/HostnameConflict` - `Conflicted/False/NoConflicts` - `OverlappingTLSConfig/True/OverlappingHostnames` ### Custom Policies | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |-----------------|--------------------|------------------------|---------------------------------------|-------------|---------------------| | Custom policies | N/A | N/A | Supported | N/A | N/A | Custom policies are NGINX Gateway Fabric-specific CRDs (Custom Resource Definitions) that support features such as tracing, and client connection settings. These important data-plane features are not part of the Gateway API specifications. While these CRDs are not part of the Gateway API, the mechanism to attach them to Gateway API resources is part of the Gateway API. See the [Policy Attachment documentation](https://gateway-api.sigs.k8s.io/references/policy-attachment/). See the [custom policies](/ngf/overview/custom-policies.md) document for more information. -------------------------------------------------------------------------------- title: "Get started" url: https://docs.nginx.com/nginx-gateway-fabric/get-started/index.md -------------------------------------------------------------------------------- **Note:** This document is for trying out NGINX Gateway Fabric, and not intended for a production environment. For standard deployments, you should read the [Install NGINX Gateway Fabric](/ngf/install/) section. This is a guide for getting started with NGINX Gateway Fabric. It explains how to: - Set up a [kind (Kubernetes in Docker)](https://kind.sigs.k8s.io/) cluster - Install [NGINX Gateway Fabric](https://github.com/nginx/nginx-gateway-fabric) with [NGINX](https://nginx.org/) - Test NGINX Gateway Fabric with an example application By following the steps in order, you will finish with a functional NGINX Gateway Fabric cluster. ## Before you begin To complete this guide, you need the following prerequisites installed: - [Go 1.16](https://go.dev/dl/) or newer, which is used by kind - [Docker](https://docs.docker.com/get-started/get-docker/), for creating and managing containers - [kind](https://kind.sigs.k8s.io/#installation-and-usage), which allows for running a local Kubernetes cluster using Docker - [kubectl](https://kubernetes.io/docs/tasks/tools/), which provides a command line interface (CLI) for interacting with Kubernetes clusters - [Helm 3.0](https://helm.sh/docs/intro/install/) or newer to install NGINX Gateway Fabric - [curl](https://curl.se/), to test the example application ## Set up a kind cluster Create the file _cluster-config.yaml_ with the following contents, noting the highlighted lines: ```yaml {linenos=true, hl_lines=[6, 9]} apiVersion: kind.x-k8s.io/v1alpha4 kind: Cluster nodes: - role: control-plane extraPortMappings: - containerPort: 31437 hostPort: 8080 protocol: TCP - containerPort: 30478 hostPort: 8443 protocol: TCP ``` **Note:** The _containerPort_ value is used to later configure a _NodePort_. Run the following command: ```shell kind create cluster --config cluster-config.yaml ``` #### Example output ```text Creating cluster "kind" ... ✓ Ensuring node image (kindest/node:v1.31.0) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-kind" You can now use your cluster with: kubectl cluster-info --context kind-kind Thanks for using kind! 😊 ``` **Note:** If you have cloned [the NGINX Gateway Fabric repository](https://github.com/nginx/nginx-gateway-fabric/tree/main), you can also create a kind cluster from the root folder with the following _make_ command: ```shell make create-kind-cluster ``` ## Install NGINX Gateway Fabric ### Add Gateway API resources Use `kubectl` to add the API resources for NGINX Gateway Fabric with the following command: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl apply -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created ``` ### Install the Helm chart Use `helm` to install NGINX Gateway Fabric, specifying the NodePort configuration that will be set on the NGINX Service when it is provisioned: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.service.type=NodePort --set-json 'nginx.service.nodePorts=[{"port":31437,"listenerPort":80}, {"port":30478,"listenerPort":8443}]' ``` **Note:** The port value should equal the _containerPort_ value from _cluster-config.yaml_ [when you created the kind cluster](#set-up-a-kind-cluster). The _listenerPort_ value will match the port that we expose in the Gateway listener. #### Example output ```text NAME: ngf LAST DEPLOYED: Tue Apr 29 14:45:14 2025 NAMESPACE: nginx-gateway STATUS: deployed REVISION: 1 TEST SUITE: None ``` ## Create an example application In the previous section, you deployed NGINX Gateway Fabric to a local cluster. This section shows you how to deploy a simple web application to test that NGINX Gateway Fabric works. **Note:** The YAML code in the following sections can be found in the [cafe-example folder](https://github.com/nginx/nginx-gateway-fabric/tree/main/examples/cafe-example) of the GitHub repository. ### Create the application resources Run the following command to create the file _cafe.yaml_, which is then used to deploy the *coffee* application to your cluster: ```yaml cat < cafe.yaml apiVersion: apps/v1 kind: Deployment metadata: name: coffee spec: replicas: 1 selector: matchLabels: app: coffee template: metadata: labels: app: coffee spec: containers: - name: coffee image: nginxdemos/nginx-hello:plain-text ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: coffee spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: coffee --- apiVersion: apps/v1 kind: Deployment metadata: name: tea spec: replicas: 1 selector: matchLabels: app: tea template: metadata: labels: app: tea spec: containers: - name: tea image: nginxdemos/nginx-hello:plain-text ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: tea spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: tea EOF kubectl apply -f cafe.yaml ``` #### Example output ```text deployment.apps/coffee created service/coffee created deployment.apps/tea created service/tea created ``` Verify that the new pods are in the `default` namespace: ```shell kubectl get pods ``` #### Example output ```text NAME READY STATUS RESTARTS AGE coffee-676c9f8944-k2bmd 1/1 Running 0 9s tea-6fbfdcb95d-9lhbj 1/1 Running 0 9s ``` ### Create Gateway and HTTPRoute resources Run the following command to create the file _gateway.yaml_, which is then used to deploy a Gateway to your cluster: ```yaml cat < gateway.yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: gateway spec: gatewayClassName: nginx listeners: - name: http port: 80 protocol: HTTP hostname: "*.example.com" EOF kubectl apply -f gateway.yaml ``` Verify that the NGINX deployment has been provisioned: ```shell kubectl get pods ``` #### Example output ```text NAME READY STATUS RESTARTS AGE coffee-676c9f8944-k2bmd 1/1 Running 0 31s gateway-nginx-66b5d78f8f-4fmtb 1/1 Running 0 13s tea-6fbfdcb95d-9lhbj 1/1 Running 0 31s ``` Run the following command to create the file _cafe-routes.yaml_. It is then used to deploy two *HTTPRoute* resources in your cluster: one each for _/coffee_ and _/tea_. ```yaml cat < cafe-routes.yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: coffee spec: parentRefs: - name: gateway sectionName: http hostnames: - "cafe.example.com" rules: - matches: - path: type: PathPrefix value: /coffee backendRefs: - name: coffee port: 80 --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: tea spec: parentRefs: - name: gateway sectionName: http hostnames: - "cafe.example.com" rules: - matches: - path: type: Exact value: /tea backendRefs: - name: tea port: 80 EOF kubectl apply -f cafe-routes.yaml ``` ### Verify the configuration You can check that all of the expected services are available using `kubectl get`: ```shell kubectl get services ``` #### Example output ```text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE coffee ClusterIP 10.96.206.93 80/TCP 2m2s gateway-nginx NodePort 10.96.157.168 80:31437/TCP 104s kubernetes ClusterIP 10.96.0.1 443/TCP 142m tea ClusterIP 10.96.43.183 80/TCP 2m2s ``` You can also use `kubectl describe` on the new resources to check their status: ```shell kubectl describe httproutes ``` #### Example output ```text Name: coffee Namespace: default Labels: Annotations: API Version: gateway.networking.k8s.io/v1 Kind: HTTPRoute Metadata: Creation Timestamp: 2025-04-29T19:06:31Z Generation: 1 Resource Version: 12285 UID: c8055a74-b4c6-442f-b3fb-350fb88b2a7c Spec: Hostnames: cafe.example.com Parent Refs: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Section Name: http Rules: Backend Refs: Group: Kind: Service Name: coffee Port: 80 Weight: 1 Matches: Path: Type: PathPrefix Value: /coffee Status: Parents: Conditions: Last Transition Time: 2025-04-29T19:06:31Z Message: The route is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2025-04-29T19:06:31Z Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Controller Name: gateway.nginx.org/nginx-gateway-controller Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Namespace: default Section Name: http Events: Name: tea Namespace: default Labels: Annotations: API Version: gateway.networking.k8s.io/v1 Kind: HTTPRoute Metadata: Creation Timestamp: 2025-04-29T19:06:31Z Generation: 1 Resource Version: 12284 UID: 55aa0ab5-9b1c-4028-9bb5-4903f05bb998 Spec: Hostnames: cafe.example.com Parent Refs: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Section Name: http Rules: Backend Refs: Group: Kind: Service Name: tea Port: 80 Weight: 1 Matches: Path: Type: Exact Value: /tea Status: Parents: Conditions: Last Transition Time: 2025-04-29T19:06:31Z Message: The route is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2025-04-29T19:06:31Z Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Controller Name: gateway.nginx.org/nginx-gateway-controller Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Namespace: default Section Name: http Events: ``` ```shell kubectl describe gateways ``` #### Example output ```text Name: gateway Namespace: default Labels: Annotations: API Version: gateway.networking.k8s.io/v1 Kind: Gateway Metadata: Creation Timestamp: 2025-04-29T19:05:01Z Generation: 1 Resource Version: 12286 UID: 0baa6e15-55e0-405a-9e7c-de22472fc3ad Spec: Gateway Class Name: nginx Listeners: Allowed Routes: Namespaces: From: Same Hostname: *.example.com Name: http Port: 80 Protocol: HTTP Status: Addresses: Type: IPAddress Value: 10.96.157.168 Conditions: Last Transition Time: 2025-04-29T19:06:31Z Message: Gateway is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2025-04-29T19:06:31Z Message: Gateway is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed Listeners: Attached Routes: 2 Conditions: Last Transition Time: 2025-04-29T19:06:31Z Message: Listener is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2025-04-29T19:06:31Z Message: Listener is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed Last Transition Time: 2025-04-29T19:06:31Z Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Last Transition Time: 2025-04-29T19:06:31Z Message: No conflicts Observed Generation: 1 Reason: NoConflicts Status: False Type: Conflicted Name: http Supported Kinds: Group: gateway.networking.k8s.io Kind: HTTPRoute Group: gateway.networking.k8s.io Kind: GRPCRoute Events: ``` ## Test NGINX Gateway Fabric By configuring the cluster with the port `31437`, there is implicit port forwarding from your local machine to NodePort, allowing for direct communication to the NGINX Gateway Fabric service. You can use `curl` to test the new services by targeting the hostname (_cafe.example.com_) with the _/coffee_ and _/tea_ paths: ```shell curl --resolve cafe.example.com:8080:127.0.0.1 http://cafe.example.com:8080/coffee ``` #### Example output ```text Server address: 10.244.0.16:8080 Server name: coffee-676c9f8944-k2bmd Date: 29/Apr/2025:19:08:21 +0000 URI: /coffee Request ID: f34e138922171977a79b1b0d0395b97e ``` ```shell curl --resolve cafe.example.com:8080:127.0.0.1 http://cafe.example.com:8080/tea ``` #### Example output ```text Server address: 10.244.0.17:8080 Server name: tea-6fbfdcb95d-9lhbj Date: 29/Apr/2025:19:08:31 +0000 URI: /tea Request ID: 1b5c8f3a4532ea7d7510cf14ffeb27af ``` ## Next steps - [Install NGINX Gateway Fabric](/ngf/install/), for additional ways to install NGINX Gateway Fabric - [Traffic management](/ngf/traffic-management/), for more in-depth traffic management configuration - [How-to guides](/ngf/how-to/), for configuring your cluster -------------------------------------------------------------------------------- title: "Get started with F5 WAF for NGINX (HTTP)" url: https://docs.nginx.com/nginx-gateway-fabric/waf-integration/get-started-http/index.md -------------------------------------------------------------------------------- This guide walks through the complete flow of protecting traffic with F5 WAF for NGINX: deploy a sample application, compile a WAF policy, apply it to a Gateway, and verify that attacks are blocked. For an overview of WAF concepts and architecture, see [F5 WAF for NGINX overview](/ngf/waf-integration/overview.md). --- ## Before you begin - [Install](/ngf/install/helm.md) NGINX Gateway Fabric using the **NGINX Plus with WAF** tab. This sets the WAF-enabled NGINX Plus image and enables WAF globally. - Have a valid F5 WAF for NGINX subscription. F5 WAF for NGINX is a separate add-on to NGINX Plus and is not included with the NGINX Plus license. - Have NGINX Gateway Fabric configured with an `imagePullSecret` for the NGINX private container registry (`private-registry.nginx.com`), either through Helm values or deployment manifests. When a Gateway is deployed, NGINX Gateway Fabric automatically creates the registry secret in the Gateway's namespace with the naming convention `-nginx-. The bundle server Deployment in this guide references the same secret for pulling the F5 WAF compiler image, be sure to update the secret name to match your environment. --- ## Deploy the sample application Deploy the `customers` and `tea` sample applications. The `customers` app is configured to return a response containing fake sensitive data (credit card number and SSN), which is used later to demonstrate data guard masking: ```yaml kubectl apply -f - <.svc.cluster.local/attack-signatures-blocking.tgz`. --- ## Verify WAF protection ### Verify the WAF containers are running Verify that the NGINX Pod has all three containers running: ```shell kubectl get pods -l app.kubernetes.io/name=gateway-nginx ``` Each NGINX Pod should show `3/3` in the `READY` column, indicating the main NGINX container, `waf-enforcer`, and `waf-config-mgr` are all running: ```text NAME READY STATUS RESTARTS AGE gateway-nginx-7f9b8d6c4d-xxxxx 3/3 Running 0 2m ``` If a container is not starting, check its logs: ```shell kubectl logs -c nginx kubectl logs -c waf-enforcer kubectl logs -c waf-config-mgr ``` ### Check WAFPolicy status Verify the WAFPolicy has been accepted and programmed: ```shell kubectl describe wafpolicy gateway-base-protection ``` Look for three conditions in the output: ```text Status: Conditions: Message: The Policy is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Message: Policy is programmed in the data plane Observed Generation: 1 Reason: Programmed Status: True Type: Programmed ``` If any condition is `False`, the message field describes the problem. See [Troubleshoot WAFPolicy status](/ngf/waf-integration/troubleshooting.md) for guidance. ### Test WAF protection Confirm the Gateway was assigned an IP address and reports a `Programmed=True` status with `kubectl describe`: ```shell kubectl describe gateways.gateway.networking.k8s.io gateway ``` ```text Addresses: Type: IPAddress Value: 192.0.2.1 ``` Save the public IP address and port(s) of the Gateway into shell variables: ```text GW_IP=XXX.YYY.ZZZ.III GW_PORT= ``` **Verify normal traffic flows.** Send a request to the `customers` route — the response contains the fake sensitive data from the `customers` backend: **Note:** If you have a DNS record allocated for `cafe.example.com`, you can send the request directly to that hostname, without needing to resolve. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/customers ``` ```text Customer List: Name: John Doe Credit Card: 4111-1111-1111-1111 SSN: 123-45-6789 ``` The sensitive data passes through because the gateway-level `attack-signatures-blocking` policy only inspects inbound requests for attack patterns — it does not mask outbound response data. **Verify attacks are blocked.** Send a request with a cross-site scripting (XSS) payload: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP "http://cafe.example.com:$GW_PORT/customers?x=" ``` The WAF detects the attack signature and rejects the request: ```text Request Rejected ... ``` **Verify the `tea` route is also protected.** Since the policy targets the Gateway, all attached routes inherit protection: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP "http://cafe.example.com:$GW_PORT/tea?x=" ``` ```text Request Rejected ... ``` **Note:** The exact blocking response depends on your WAF policy configuration. Check the security log (stderr in this example) for a corresponding blocked event using `kubectl logs -c waf-enforcer`. --- ## Apply a route-level override In the previous step, you saw that the `customers` route returns sensitive data (credit card numbers and SSNs) in the response body. The gateway-level `attack-signatures-blocking` policy blocks inbound attacks, but does not inspect outbound responses. To protect sensitive data in responses, apply a **data guard** policy as a route-level override on the `customers` route. This policy masks credit card numbers and Social Security numbers in response bodies. The `dataguard-blocking` bundle was already compiled by the bundle server init container at startup — no additional compilation is needed. ### Apply the route-level WAFPolicy ```yaml kubectl apply -f - < **Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. Use a regular NGINX Plus instance JWT. ### Create the Docker Registry Secret **Note:** If you would rather pull the NGINX Plus image and push to a private registry, you can skip this specific step and instead follow [this step](/ngf/install/nginx-plus.md#pull-an-image-for-local-use). If the `nginx-gateway` namespace does not yet exist, create it: ```shell kubectl create namespace nginx-gateway ``` Create a Kubernetes `docker-registry` secret type using the contents of the JWT as the username and `none` for password (as the password is not used). The name of the docker server is `private-registry.nginx.com`. ```shell kubectl create secret docker-registry nginx-plus-registry-secret --docker-server=private-registry.nginx.com --docker-username= --docker-password=none -n nginx-gateway ``` It is important that the `--docker-username=` contains the contents of the token and is not pointing to the token itself. When you copy the contents of the JWT, ensure there are no additional characters such as extra whitespaces. This can invalidate the token, causing 401 errors when trying to authenticate to the registry. ### Create the NGINX Plus Secret Place the JWT in a file called `license.jwt`. Create a Kubernetes Secret using the contents of the JWT file. ```shell kubectl create secret generic nplus-license --from-file license.jwt -n nginx-gateway ``` You can now delete the `license.jwt` file. If you need to update the JWT at any time, update the `license.jwt` field in the Secret using `kubectl edit` and apply the changes. **Note:** For more information on why this is needed and additional configuration options, including how to report to NGINX Instance Manager instead, see the [NGINX Plus Image and JWT Requirement](/ngf/install/nginx-plus.md) document. ## Deploy NGINX Gateway Fabric ### Installing the Gateway API resources **Note:** The [Gateway API resources](https://github.com/kubernetes-sigs/gateway-api) from the standard channel must be installed before deploying NGINX Gateway Fabric. If they are already installed in your cluster, please ensure they are the correct version as supported by the NGINX Gateway Fabric - [see the Technical Specifications](https://github.com/nginx/nginx-gateway-fabric/blob/v/README.md#technical-specifications). To install the Gateway API resources, run the following: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl apply -f - ``` **Note:** If you plan to use the `edge` version of NGINX Gateway Fabric, you can replace the version in `ref` with `main`, for example `ref=main`. Alternatively, you can install the Gateway API resources from the experimental channel. Installing Gateway API resources from the experimental channel includes everything in the standard release channel plus additional experimental resources and fields. NGINX Gateway Fabric currently supports a subset of the additional features provided by the experimental channel. To install from the experimental channel, run the following: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl apply -f - ``` **Note:** To learn more about what Gateway API resources NGINX Gateway Fabric currently supports, visit our [Gateway API Compatibility](/ngf/overview/gateway-api-compatibility.md) document. ### Install from the OCI registry The following steps install NGINX Gateway Fabric directly from the OCI helm registry. If you prefer, you can [install from sources](#install-from-sources) instead. #### NGINX To install the latest stable release of NGINX Gateway Fabric in the **nginx-gateway** namespace, run the following command: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway ``` #### NGINX Plus **Note:** If applicable, replace the F5 Container registry `private-registry.nginx.com` with your internal registry for your NGINX Plus image, and replace `nginx-plus-registry-secret` with your Secret name containing the registry credentials. If your NGINX Plus JWT Secret has a different name than the default `nplus-license`, then define that name using the `nginx.usage.secretName` flag. To install the latest stable release of NGINX Gateway Fabric in the **nginx-gateway** namespace, run the following command: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --set nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus --set nginx.plus=true --set nginx.imagePullSecret=nginx-plus-registry-secret -n nginx-gateway ``` #### NGINX Plus with WAF **Note:** If applicable, replace the F5 Container registry `private-registry.nginx.com` with your internal registry, and replace `nginx-plus-registry-secret` with your Secret name containing the registry credentials. If your NGINX Plus JWT Secret has a different name than the default `nplus-license`, then define that name using the `nginx.usage.secretName` flag. To install the latest stable release of NGINX Gateway Fabric with F5 WAF for NGINX enabled globally across all Gateways, run the following command: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --set nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus-f5waf --set nginx.plus=true --set nginx.config.waf.enable=true --set nginx.imagePullSecret=nginx-plus-registry-secret --create-namespace -n nginx-gateway ``` `ngf` is the name of the release, and can be changed to any name you want. This name is added as a prefix to the Deployment name. If you want the latest version from the **main** branch, add `--version 0.0.0-edge` to your install command. To wait for the Deployment to be ready, you can either add the `--wait` flag to the `helm install` command, or run the following after installing: ```shell kubectl wait --timeout=5m -n nginx-gateway deployment/ngf-nginx-gateway-fabric --for=condition=Available ``` ### Install from sources {#install-from-sources} If you prefer to install directly from sources, instead of through the OCI helm registry, use the following steps. ```shell helm pull oci://ghcr.io/nginx/charts/nginx-gateway-fabric --untar cd nginx-gateway-fabric ``` For the latest version from the **main** branch, add _--version 0.0.0-edge_ to your pull command. #### NGINX To install the chart into the **nginx-gateway** namespace, run the following command: ```shell helm install ngf . --create-namespace -n nginx-gateway ``` #### NGINX Plus **Note:** If applicable, replace the F5 Container registry `private-registry.nginx.com` with your internal registry for your NGINX Plus image, and replace `nginx-plus-registry-secret` with your Secret name containing the registry credentials. If your NGINX Plus JWT Secret has a different name than the default `nplus-license`, then define that name using the `nginx.usage.secretName` flag. To install the chart into the **nginx-gateway** namespace, run the following command: ```shell helm install ngf . --set nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus --set nginx.plus=true --set nginx.imagePullSecret=nginx-plus-registry-secret -n nginx-gateway ``` #### NGINX Plus with WAF **Note:** If applicable, replace the F5 Container registry `private-registry.nginx.com` with your internal registry, and replace `nginx-plus-registry-secret` with your Secret name containing the registry credentials. If your NGINX Plus JWT Secret has a different name than the default `nplus-license`, then define that name using the `nginx.usage.secretName` flag. To install the chart into the **nginx-gateway** namespace with F5 WAF for NGINX enabled globally across all Gateways, run the following command: ```shell helm install ngf . --set nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus-f5waf --set nginx.plus=true --set nginx.config.waf.enable=true --set nginx.imagePullSecret=nginx-plus-registry-secret -n nginx-gateway ``` `ngf` is the name of the release, and can be changed to any name you want. This name is added as a prefix to the Deployment name. To wait for the Deployment to be ready, you can either add the `--wait` flag to the `helm install` command, or run the following after installing: ```shell kubectl wait --timeout=5m -n nginx-gateway deployment/ngf-nginx-gateway-fabric --for=condition=Available ``` ### Custom installation options #### Service type By default, the NGINX Gateway Fabric control plane deploys a LoadBalancer Service when a Gateway is created. To use a NodePort Service instead: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.service.type=NodePort ``` #### Experimental features We support a subset of the additional features provided by the Gateway API experimental channel. To enable the experimental features of Gateway API which are supported by NGINX Gateway Fabric: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginxGateway.gwAPIExperimentalFeatures.enable=true ``` **Note:** Requires the Gateway APIs installed from the experimental channel. #### Watch subset of namespaces By default, the NGINX Gateway Fabric controller watches resources in all namespaces. To restrict the list of namespaces, you can specify `--set "nginxGateway.watchNamespaces={x,y,z}"` with your namespace names when installing. #### Examples You can find several examples of configuration options of the `values.yaml` file in the [helm examples](https://github.com/nginx/nginx-gateway-fabric/tree/v/examples/helm) directory. ### Access NGINX Gateway Fabric When NGINX Gateway Fabric is installed, it provisions a ClusterIP Service used only for internal communication between the control plane and data planes. To deploy NGINX itself and get a LoadBalancer Service, you should follow the [Deploy a Gateway for data plane instances](/ngf/install/deploy-data-plane.md) instructions. ## Uninstall NGINX Gateway Fabric Follow these steps to uninstall NGINX Gateway Fabric and Gateway API from your Kubernetes cluster: 1. **Uninstall NGINX Gateway Fabric:** - To uninstall NGINX Gateway Fabric, run: ```shell helm uninstall ngf -n nginx-gateway ``` If needed, replace `ngf` with your chosen release name. 2. **Remove namespace and CRDs:** - To remove the **nginx-gateway** namespace and its custom resource definitions (CRDs), run: ```shell kubectl delete ns nginx-gateway kubectl delete -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/deploy/crds.yaml ``` 3. **Remove the Gateway API resources:** - **Note:** This step will remove all corresponding custom resources in your entire cluster, across every namespace. Ensure you don't have any custom resources you need to keep, and confirm that there are no other Gateway API implementations active in your cluster. To uninstall the Gateway API resources, run the following: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl delete -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io "backendtlspolicies.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "gateways.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "httproutes.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "referencegrants.gateway.networking.k8s.io" deleted ``` If you installed the Gateway APIs from the experimental channel, run the following instead: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl delete -f - ``` ## Next steps - [Deploy a Gateway for data plane instances](/ngf/install/deploy-data-plane.md) - [Routing traffic to applications](/ngf/traffic-management/basic-routing.md) For a full list of the Helm Chart configuration parameters, read [the NGINX Gateway Fabric Helm Chart](https://github.com/nginx/nginx-gateway-fabric/blob/v/charts/nginx-gateway-fabric/README.md#configuration). -------------------------------------------------------------------------------- title: "Monitoring with Prometheus and Grafana" url: https://docs.nginx.com/nginx-gateway-fabric/monitoring/prometheus/index.md -------------------------------------------------------------------------------- This document describes how to monitor NGINX Gateway Fabric using Prometheus and Grafana. It explains installation and configuration, as well as what metrics are available. ## Overview NGINX Gateway Fabric metrics are displayed in [Prometheus](https://prometheus.io/) format. These metrics are served through a metrics server orchestrated by the controller-runtime package on HTTP port `9113`. When installed, Prometheus automatically scrapes this port and collects metrics. [Grafana](https://grafana.com/) can be used for rich visualization of these metrics. **Note:** Metrics are served over HTTP by default. Enabling HTTPS will secure the metrics endpoint with a self-signed certificate. When using HTTPS, adjust the Prometheus Pod scrape settings by adding the `insecure_skip_verify` flag to handle the self-signed certificate. For further details, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config). ## Installing Prometheus and Grafana **Note:** These installations are for demonstration purposes and have not been tuned for a production environment. ### Prometheus ```shell helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus prometheus-community/prometheus -n monitoring --create-namespace --set server.global.scrape_interval=15s ``` Once running, you can access the Prometheus dashboard by using port-forwarding in the background: ```shell kubectl port-forward -n monitoring svc/prometheus-server 9090:80 & ``` Visit [http://127.0.0.1:9090](http://127.0.0.1:9090) to view the dashboard. ### Grafana ```shell helm repo add grafana-community https://grafana-community.github.io/helm-charts helm repo update helm install grafana grafana-community/grafana -n monitoring --create-namespace ``` Once running, you can access the Grafana dashboard by using port-forwarding in the background: ```shell kubectl port-forward -n monitoring svc/grafana 3000:80 & ``` Visit [http://127.0.0.1:3000](http://127.0.0.1:3000) to view the Grafana UI. The username for login is `admin`. The password can be acquired by running: ```shell kubectl get secret -n monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` #### Configuring Grafana In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Prometheus service (`http://prometheus-server.monitoring.svc`) as a data source. Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. - [Download: ngf-grafana-dashboard.json](ngf/grafana-dashboard.json) ## Available metrics in NGINX Gateway Fabric NGINX Gateway Fabric provides a variety of metrics for monitoring and analyzing performance. These metrics are categorized as follows: ### NGINX/NGINX Plus metrics NGINX metrics include NGINX-specific data such as the total number of accepted client connections. These metrics are collected through NGINX Agent and are reported by each NGINX Pod. NGINX Gateway Fabric currently supports a subset of all metrics available through NGINX OSS and Plus. Listed below are the supported metrics along with a small accompanying description. Metrics provided by NGINX Open Source include: - `nginx_http_connection_count_connections`: The current number of connections. - `nginx_http_connections_total`: The total number of connections, since NGINX was last started or reloaded. - `nginx_http_request_count_requests`: The total number of client requests received, since the last collection interval. - `nginx_http_requests_total`: The total number of client requests received, since NGINX was last started or reloaded. In addition to the previous metrics provided by NGINX Open Source, NGINX Plus includes: - `nginx_config_reloads_total`: The total number of NGINX config reloads. - `nginx_http_response_count_responses`: The total number of HTTP responses sent to clients since the last collection interval, grouped by status code range. - `nginx_http_response_status_responses_total`: The total number of responses since NGINX was last started or reloaded, grouped by status code range. - `nginx_http_request_discarded_requests_total`: The total number of requests completed without sending a response. - `nginx_http_request_processing_count_requests`: The number of client requests that are currently being processed. - `nginx_http_request_byte_io_bytes_total`: The total number of HTTP byte IO. - `nginx_http_upstream_keepalive_count_connections`: The current number of idle keepalive connections per HTTP upstream. - `nginx_http_upstream_peer_connection_count_connections`: The average number of active connections per HTTP upstream peer. - `nginx_http_upstream_peer_byte_io_bytes_total`: The total number of byte IO per HTTP upstream peer. - `nginx_http_upstream_peer_count_peers`: The current count of peers on the HTTP upstream grouped by state. - `nginx_http_upstream_peer_fails_attempts_total`: The total number of unsuccessful attempts to communicate with the HTTP upstream peer. - `nginx_http_upstream_peer_header_time_milliseconds`: The average time to get the response header from the HTTP upstream peer. - `nginx_http_upstream_peer_health_checks_requests_total`: The total number of health check requests made to a HTTP upstream peer. - `nginx_http_upstream_peer_requests_total`: The total number of client requests forwarded to the HTTP upstream peer. - `nginx_http_upstream_peer_response_time_milliseconds`: The average time to get the full response from the HTTP upstream peer. - `nginx_http_upstream_peer_responses_total`: The total number of responses obtained from the HTTP upstream peer grouped by status range. - `nginx_http_upstream_peer_state_is_deployed`: Current state of an upstream peer in deployment. - `nginx_http_upstream_peer_unavailables_requests_total`: Number of times the server became unavailable for client requests (“unavail”). - `nginx_http_upstream_queue_limit_requests`: The maximum number of requests that can be in the queue at the same time. - `nginx_http_upstream_queue_overflows_responses_total`: The total number of requests rejected due to the queue overflow. - `nginx_http_upstream_queue_usage_requests`: The current number of requests in the queue. - `nginx_http_upstream_zombie_count_is_deployed`: The current number of upstream peers removed from the group but still processing active client requests. - `nginx_slab_page_free_pages`: The current number of free memory pages. - `nginx_slab_page_usage_pages`: The current number of used memory pages. - `nginx_slab_slot_allocations_total`: The number of attempts to allocate memory of specified size. - `nginx_slab_slot_free_slots`: The current number of free memory slots. - `nginx_slab_slot_usage_slots`: The current number of used memory slots. - `nginx_ssl_certificate_verify_failures_certificates_total`: The total number of SSL certificate verification failures. - `nginx_ssl_handshakes_total`: The total number of SSL handshakes. ### NGINX Gateway Fabric metrics Metrics specific to NGINX Gateway Fabric include: - `event_batch_processing_milliseconds`: Time in milliseconds to process batches of Kubernetes events. All these metrics are under the `nginx_gateway_fabric` namespace and include a `class` label set to the GatewayClass of NGINX Gateway Fabric. For example, `nginx_gateway_fabric_event_batch_processing_milliseconds_sum{class="nginx"}`. ### Controller-runtime metrics Provided by the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) library, these metrics include: - General resource usage like CPU and memory. - Go runtime metrics such as the number of Go routines, garbage collection duration, and Go version. - Controller-specific metrics, including reconciliation errors per controller, length of the reconcile queue, and reconciliation latency. ## Change the default metrics configuration You can configure monitoring metrics for NGINX Gateway Fabric using Helm or Manifests. ### Using Helm If you're setting up NGINX Gateway Fabric with Helm, you can adjust the `metrics.*` parameters to fit your needs. For detailed options and instructions, see the [Helm README](https://github.com/nginx/nginx-gateway-fabric/blob/v/charts/nginx-gateway-fabric/README.md). ### Using Kubernetes manifests For setups using Kubernetes manifests, change the metrics configuration by editing the NGINX Gateway Fabric manifest that you want to deploy. You can find some examples in the [deploy](https://github.com/nginx/nginx-gateway-fabric/tree/v/deploy) directory. #### Disabling metrics If you need to disable metrics: 1. Set the `-metrics-disable` [command-line argument](/ngf/reference/cli-help.md) to `true` in the NGINX Gateway Fabric Pod's configuration. Remove any other `-metrics-*` arguments. 2. In the Pod template for NGINX Gateway Fabric, delete the metrics port entry from the container ports list: ```yaml - name: metrics containerPort: 9113 ``` 3. Also, remove the following annotations from the NGINX Gateway Fabric Pod template: ```yaml annotations: prometheus.io/scrape: "true" prometheus.io/port: "9113" ``` #### Changing the default port To change the default port for metrics: 1. Update the `-metrics-port` [command-line argument](/ngf/reference/cli-help.md) in the NGINX Gateway Fabric Pod's configuration to your chosen port number. 2. In the Pod template, change the metrics port entry to reflect the new port: ```yaml - name: metrics containerPort: ``` 3. Modify the `prometheus.io/port` annotation in the Pod template to match the new port: ```yaml annotations: <...> prometheus.io/port: "" <...> ``` #### Enabling HTTPS for metrics For enhanced security with HTTPS: 1. Enable HTTPS security by setting the `-metrics-secure-serving` [command-line argument](/ngf/reference/cli-help.md) to `true` in the NGINX Gateway Fabric Pod's configuration. 2. Add an HTTPS scheme annotation to the Pod template: ```yaml annotations: <...> prometheus.io/scheme: "https" <...> ``` -------------------------------------------------------------------------------- title: "Use Manifests to install NGINX Gateway Fabric with NGINX Plus" url: https://docs.nginx.com/nginx-gateway-fabric/install/manifests/plus/index.md -------------------------------------------------------------------------------- This page describes how to use Manifests to install NGINX Gateway Fabric with NGINX Plus. It explains how to install the Gateway API resources and add certificates for secure authentication, then deploy NGINX Gateway Fabric and its custom resource definitions. By following these instructions, you will finish with a functional NGINX Gateway Fabric instance for your Kubernetes cluster. **Note:** To learn which Gateway API resources NGINX Gateway Fabric currently supports, view the [Gateway API Compatibility](/ngf/overview/gateway-api-compatibility.md) topic. To install an experimental NGINX Gateway Fabric version view the [Use Manifests to install NGINX Gateway Fabric with NGINX Plus](/ngf/install/manifests/plus-experimental.md) topic. ## Before you begin To complete this guide, you will need the following pre-requisites: - An active NGINX Plus subscription (Purchased or trial) - [A supported Kubernetes version](/ngf/overview/technical-specifications.md) - A functional Kubernetes cluster ## Download your JSON web token 1. Log in to [MyF5](https://my.f5.com/manage/s/). 2. Go to **My Products & Plans** > **Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. Use a regular NGINX Plus instance JWT. ## Create license and registry secrets First, create the _nginx-gateway_ namespace, which is used by the Manifest files by default: ```shell kubectl create namespace nginx-gateway ``` **Note:** The commands in the rest of this document should be run in the same directory as your **license.jwt** file. JWTs are sensitive information and should be stored securely. Delete them after use to prevent unauthorized access. Once you have obtained your license JWT, create a Kubernetes secret using `kubectl create`: ```shell kubectl create -n nginx-gateway secret generic nplus-license --from-file license.jwt ``` Then create another Kubernetes secret to allow interactions with the F5 registry: ```shell kubectl create -n nginx-gateway secret docker-registry nginx-plus-registry-secret \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat license.jwt) \ --docker-password=none ``` You can verify the creation of the secrets using `kubectl get`: ```shell kubectl get -n nginx-gateway secrets ``` #### Example output ```text NAME TYPE DATA AGE nginx-plus-registry-secret kubernetes.io/dockerconfigjson 1 8s nplus-license Opaque 1 21s ``` ## Install the Gateway API resources **Note:** If you have already installed Gateway API resources in your cluster, ensure they are a version [supported by NGINX Gateway Fabric](/ngf/overview/technical-specifications.md) To install the Gateway API resources, use `kubectl kustomize`: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl apply -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created ``` ## Install cert-manager Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ## Add certificates for secure authentication **Note:** These steps use a self-signed issuer, which should not be used in production environments. For production environments, you should use a real [CA issuer](https://cert-manager.io/docs/configuration/ca/). First, create a CA (certificate authority) issuer: ```yaml kubectl apply -f - <= v1.18.0, the default value changed from `Never` to `Always`. certificate.cert-manager.io/nginx-gateway-ca created issuer.cert-manager.io/nginx-gateway-issuer created ``` You will then need to create a server certificate for the NGINX Gateway Fabric control plane (server): **Note:** The default service name is _nginx-gateway_, and the namespace is _nginx-gateway_, so the `dnsNames` value should be `nginx-gateway.nginx-gateway.svc`. This value becomes the name of the NGINX Gateway Fabric control plane service. ```yaml {hl_lines=[13]} kubectl apply -f - <>gw: Send request and present cert for validation gw->>client: Present certificate from Secret: cafe-secret client->>client: Validate Gateway's certificate gw->>gw: Validate client certificate with CA Secret gw->>app: Request to backend app-->>gw: Response gw-->>client: Response ``` ## Before you begin Before starting, you will need: - Administrator access to a Kubernetes cluster. - [Helm](https://helm.sh/) and [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) must be installed locally. - [NGINX Gateway Fabric deployed](/ngf/install/) in the Kubernetes cluster. ## Install cert-manager Frontend TLS requires CA certificates for client certificate validation. This example uses [cert-manager](https://cert-manager.io/) to issue these certificates. Add the Helm repository: ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` Install cert-manager: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ## Create CA certificates and issuers Create a CA issuer to generate our certificates. **Note:** This example uses a `selfSigned` Issuer, which should not be used in production environments. For production environments, use a real [CA issuer](https://cert-manager.io/docs/configuration/ca/). Next, we create the following resources: 1. A self-signed issuer. 2. A CA certificate named `default-validation-ca-secret` for our **default** frontend TLS validation. 3. A CA certificate named `per-port-validation-ca-secret` for our **perPort** frontend TLS validation. 4. A CA certificate named `cafe-secret`. The HTTPS listeners on the Gateway reference this certificate and present it to the client during the TLS handshake. This is required for mutual TLS. **Note:** For the Gateway's certificate, replace `cafe.example.com` with the correct hostname for your environment ```yaml kubectl apply -f - < GW_PER_PORT= ``` ## Create HTTPRoutes Copy the YAML code below into your terminal to create HTTPRoutes to route traffic to the backend applications: ```yaml kubectl apply -f - < -- Status: Parents: Conditions: Last Transition Time: 2026-05-06T06:57:53Z Message: The Route is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2026-05-06T06:57:53Z Message: All references are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Controller Name: gateway.nginx.org/nginx-gateway-controller Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Namespace: default Section Name: https Events: ``` ## Setup configuration test To send requests to the Gateway, you must provide a valid certificate and key signed by a valid Certificate Authority (CA). Copy the following block into your terminal to create two Certificate resources. This will create two `Secret` resources with TLS certs and keys signed by the CAs created earlier in this example. **Note:** Replace `cafe.example.com` with the correct hostname for your environment ```yaml kubectl apply -f - <` with the actual name of the pod). ```shell kubectl port-forward 8765:8765 -n ``` Afterwards, use a browser to access [http://127.0.0.1:8765/dashboard.html](http://127.0.0.1:8765/dashboard.html) to view the dashboard. The dashboard will look like this: ![image](/ngf/img/nginx-plus-dashboard.png) **Note:** The [API](https://nginx.org/en/docs/http/ngx_http_api_module.html) used by the dashboard for metrics is also accessible using the `/api` path. ### Configure dashboard access through NginxProxy To access the NGINX Plus dashboard from sources than the default `127.0.0.1`, you can use the NginxProxy resource to allow access to other IP Addresses or CIDR blocks. The following example configuration allows access to the NGINX Plus dashboard from the IP Addresses `192.0.2.8` and `192.0.2.0` and the CIDR block `198.51.100.0/24`: ```yaml apiVersion: gateway.nginx.org/v1alpha1 kind: NginxProxy metadata: name: ngf-proxy-config spec: nginxPlus: allowedAddresses: - type: IPAddress value: 192.0.2.8 - type: IPAddress value: 192.0.2.0 - type: CIDR value: 198.51.100.0/24 ``` For more information on configuring the NginxProxy resource, visit the [data plane configuration](/ngf/how-to/data-plane-configuration.md) document. -------------------------------------------------------------------------------- title: "Advanced features with NGINX Plus" url: https://docs.nginx.com/nginx-gateway-fabric/overview/nginx-plus/index.md -------------------------------------------------------------------------------- NGINX Gateway Fabric can use NGINX Open Source or NGINX Plus as its data plane. [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) is the closed source, commercial version of NGINX. Using NGINX Plus as the data plane offers additional benefits compared to the open source version. ## Benefits of NGINX Plus - **Robust metrics**: A plethora of [additional Prometheus metrics](/ngf/monitoring/prometheus.md) are available. - **Live activity monitoring**: The [NGINX Plus dashboard](/ngf/monitoring/dashboard.md) shows real-time metrics and information about your server infrastructure. - **Dynamic upstream configuration**: NGINX Plus can dynamically reconfigure upstream servers when applications in Kubernetes scale up and down, preventing the need for an NGINX reload. - **Session persistence**: NGINX Plus provides support for cookie-based session persistence, allowing client requests to be consistently routed to the same upstream pod. - **Load balancing methods**: NGINX Plus provides additional latency-aware load balancing methods `random two least_time=header` and `random two least_time=last_byte` that route traffic to the server with the least average response time among two randomly selected servers. - **JWT and OIDC Authentication**: [JSON Web Token (JWT)](/ngf/traffic-security/jwt-authentication.md) and [OpenID Connect (OIDC)](/ngf/traffic-security/oidc-authentication.md) authentication support for access control and auth delegation. - **Web Application Firewall**: NGINX Plus enables integration with [F5 WAF for NGINX](/ngf/waf-integration/overview.md) (separate add-on subscription), providing enterprise-grade protection against web exploits, injection attacks, and other OWASP Top 10 threats. WAF policies are applied at the Gateway or Route level using the `WAFPolicy` custom resource. - **Support**: With an NGINX Plus license, you can take advantage of full [support](https://my.f5.com/manage/s/article/K000140156/) from NGINX, Inc. -------------------------------------------------------------------------------- title: "Configure HTTPS termination" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/https-termination/index.md -------------------------------------------------------------------------------- Learn how to terminate HTTPS traffic using NGINX Gateway Fabric. ## Overview In this guide, we will show how to configure HTTPS termination for your application, using an [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) redirect filter, secret, and [ReferenceGrant](https://gateway-api.sigs.k8s.io/api-types/referencegrant/). **Note:** To validate client certificates using mutual TLS (mTLS), see [Securing frontend client traffic using mutual TLS](/ngf/traffic-security/client-validation.md). --- ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Set up Create the **coffee** application in Kubernetes by copying and pasting the following block into your terminal: ```yaml kubectl apply -f - < 80/TCP 40s ``` ## Configure HTTPS termination and routing For HTTPS, the deployment requires a certificate and a private key stored in a Secret. The Secret lives in a separate namespace, so a ReferenceGrant is required to access it. cert-manager issues the certificate from a local self-signed CA and automatically creates the `cafe-secret` Secret. To create the **certificate** namespace, copy and paste the following into your terminal: ```yaml kubectl apply -f - < GW_HTTPS_PORT= ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. To create the httproute resources, copy and paste the following into your terminal: ```yaml kubectl apply -f - <" password: "" EOF ``` To use a Bearer Token instead, create the Secret with a `token` key: ```yaml kubectl apply -f - < EOF ``` ### Create a gateway-level WAFPolicy The following `WAFPolicy` targets the Gateway and protects all attached routes: ```yaml kubectl apply -f - < EOF ``` ### Create a WAFPolicy The following example uses `policyObjectID` to reference the policy directly. You can use `policyName` instead if you prefer to reference by name: ```yaml kubectl apply -f - <.console.ves.volterra.io namespace: default policyObjectID: pol_12345_WTHGmDEX9qnbVjQ policyVersionID: pv_Tm__12345oWmJgwxiKlHAg auth: secretRef: name: n1c-credentials securityLogs: - destination: type: syslog syslog: server: syslog-svc.default.svc.cluster.local:514 logSource: n1cSource: url: https://.console.ves.volterra.io namespace: default profileObjectID: "lp_8s8uZxLpThWwEGF7LTn_rA" auth: secretRef: name: n1c-credentials EOF ``` Replace `` with your NGINX One Console tenant hostname. The `namespace` field refers to the NGINX One Console namespace where the policy resides. **Note:** To pin a specific policy version, set `policyVersionID`. A pinned version always resolves to the same compiled bundle. Turn off polling to avoid unnecessary network requests. If you use only `policyName` or `policyObjectID` without a version pin, NGINX Gateway Fabric fetches the latest compiled bundle on each reconciliation or poll cycle. --- ## HTTP/HTTPS server Use this option when you compile F5 WAF policies using the F5 WAF compiler CLI or a CI/CD pipeline and host the resulting bundle on an HTTP/HTTPS server. For details on using the compiler, see [Build and use the compiler tool](/waf/configure/compiler.md). For a complete walkthrough including policy compilation and a bundle server deployment, see [Get started with F5 WAF for NGINX](/ngf/waf-integration/get-started-http.md). In production environments, host compiled bundles on an HTTPS server with authentication. See [Configure WAF settings](/ngf/waf-integration/configuration.md) for details on adding credentials, custom CA certificates, and checksum verification to your `policySource`. --- ## Policy Lifecycle Management (PLM) Use this option when you manage F5 WAF policies as Kubernetes resources with Policy Lifecycle Management (PLM). With PLM, you define your security posture as `APPolicy` and `APLogConf` custom resources instead of `policySource`/`logSource`. The PLM controller compiles these resources automatically and stores the resulting bundles in in-cluster storage. NGINX Gateway Fabric fetches those bundles and deploys them to the data plane. Because PLM is event-driven, you don't need a per-`WAFPolicy` credentials Secret or polling configuration. For a comparison of PLM with the other source types, see [PLM](/ngf/waf-integration/overview.md#plm-policy-lifecycle-management). For a complete walkthrough, including PLM storage setup, defining `APPolicy`/`APLogConf` resources, and applying a `WAFPolicy`, see [Get started with F5 WAF for NGINX using PLM](/ngf/waf-integration/get-started-plm.md). --- ## Policy deployment visibility NGINX Instance Manager and NGINX One Console don't show which F5 WAF policies are deployed to NGINX Gateway Fabric. Neither console shows which compiled bundle version NGINX Gateway Fabric has fetched. This is intentional. NGINX Gateway Fabric pulls compiled bundles from the management plane and deploys them directly in Kubernetes using native manifests, not through NGINX Instance Manager or NGINX One Console. This design lets you create and compile policies, then make them available to NGINX Gateway Fabric through the API, without a console-managed deployment step. F5 plans to add policy association visibility for NGINX Instance Manager and NGINX One Console in a future release. In the meantime, use `kubectl describe wafpolicy ` to check deployment status. --- ## Security event monitoring ### NGINX Instance Manager #### Connect NGINX Gateway Fabric to F5 NGINX Instance Manager Configure NGINX Gateway Fabric to connect to NGINX Instance Manager before continuing. Follow [Connect NGINX Gateway Fabric to NGINX Instance Manager](/nim/connect-kubernetes/connect-ngf.md). #### Export security logs to F5 NGINX Instance Manager The Security Monitoring dashboard in NGINX Instance Manager doesn't show which F5 WAF policies are deployed to NGINX Gateway Fabric data planes. You can still export F5 WAF security events to NGINX Instance Manager. Exporting these events gives your security operations team visibility into blocked attacks, violations, and traffic patterns directly in the dashboard. **Note:** This integration requires NGINX Instance Manager 2.23 or later. To export these events, configure a `securityLogs` entry on the `WAFPolicy` resource. The entry sends events to the syslog listener that NGINX Agent runs inside the NGINX pod. NGINX Agent's built-in OpenTelemetry collector transforms the events and exports them to NGINX Instance Manager. ```yaml kubectl apply -f - <.console.ves.volterra.io namespace: default policyName: "ngfExample" auth: secretRef: name: n1c-credentials securityLogs: - destination: type: syslog syslog: server: localhost:1514 logSource: n1cSource: url: https://.console.ves.volterra.io namespace: default profileName: "secops_dashboard" auth: secretRef: name: n1c-credentials EOF ``` The `localhost:1514` syslog destination points to NGINX Agent's OpenTelemetry collector, which runs as a sidecar in the NGINX pod. NGINX Agent forwards the security events to NGINX One Console, where they appear in the security monitoring dashboard. **Note:** The `profileName: "secops_dashboard"` log profile must exist in your NGINX One Console namespace. Events don't appear correctly in the NGINX One Console security dashboard without this profile. --- ## See also - [F5 WAF for NGINX overview](/ngf/waf-integration/overview.md) - [Get started with F5 WAF for NGINX](/ngf/waf-integration/get-started-http.md) - [Get started with F5 WAF for NGINX using PLM](/ngf/waf-integration/get-started-plm.md) - [Configure WAF settings](/ngf/waf-integration/configuration.md) - [Troubleshoot WAFPolicy status](/ngf/waf-integration/troubleshooting.md) - [Technical specifications](/ngf/overview/technical-specifications.md) - [WAFPolicy and NginxProxy API reference](/ngf/reference/api.md) -------------------------------------------------------------------------------- title: "Install NGINX Gateway Fabric with NGINX Plus" url: https://docs.nginx.com/nginx-gateway-fabric/install/nginx-plus/index.md -------------------------------------------------------------------------------- ## Overview NGINX Gateway Fabric with NGINX Plus requires a valid JSON Web Token (JWT) to download the container image from the F5 registry. From version 1.5.0, this JWT token is also required to run NGINX Plus. This requirement is part of F5’s broader licensing program and aligns with industry best practices. The JWT will streamline subscription renewals and usage reporting, helping you manage your NGINX Plus subscription more efficiently. The [telemetry](#telemetry) data we collect helps us improve our products and services to better meet your needs. The JWT is required for validating your subscription and reporting telemetry data. For environments connected to the internet, telemetry is automatically sent to F5’s licensing endpoint. In offline environments, telemetry is routed through [NGINX Instance Manager](/nim/). Usage is reported every hour and on startup whenever NGINX is reloaded. **Note:** The following Secrets should be created in the same namespace as the NGINX Gateway Fabric control plane (default: nginx-gateway). The control plane will copy these Secrets into any namespaces where NGINX gets deployed. If you need to update the Secrets, update the originals that you created in the control plane namespace, and the control plane will propagate those updates to all duplicated Secrets. ## Set up the JWT The JWT needs to be configured before deploying NGINX Gateway Fabric. The JWT will be stored in two Kubernetes Secrets: one for downloading the NGINX Plus container image, and the other for running NGINX Plus. **Note:** For security, follow these practices with JSON Web Tokens (JWTs), passwords, and shell history: 1. **JWTs:** JWTs are sensitive information. Store them securely. Delete them after use to prevent unauthorized access. 1. **Shell history:** Commands that include JWTs or passwords are recorded in the history of your shell, in plain text. Clear your shell history after running such commands. For example, if you use bash, you can delete commands in your `~/.bash_history` file. Alternatively, you can run the `history -c` command to erase your shell history. Follow these practices to help ensure the security of your system and data. ### Download the JWT from MyF5 1. Log in to [MyF5](https://my.f5.com/manage/s/). 2. Go to **My Products & Plans** > **Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. Use a regular NGINX Plus instance JWT. ### Docker Registry Secret **Note:** If you would rather pull the NGINX Plus image and push to a private registry, you can skip this specific step and instead follow [this step](/ngf/install/nginx-plus.md#pull-an-image-for-local-use). If the `nginx-gateway` namespace does not yet exist, create it: ```shell kubectl create namespace nginx-gateway ``` Create a Kubernetes `docker-registry` secret type using the contents of the JWT as the username and `none` for password (as the password is not used). The name of the docker server is `private-registry.nginx.com`. ```shell kubectl create secret docker-registry nginx-plus-registry-secret --docker-server=private-registry.nginx.com --docker-username= --docker-password=none -n nginx-gateway ``` It is important that the `--docker-username=` contains the contents of the token and is not pointing to the token itself. When you copy the contents of the JWT, ensure there are no additional characters such as extra whitespaces. This can invalidate the token, causing 401 errors when trying to authenticate to the registry. Provide the name of this Secret when installing NGINX Gateway Fabric: #### Helm Specify the Secret name using the `nginx.imagePullSecret` or `nginx.imagePullSecrets` helm value. #### Manifests Specify the Secret name in the `nginx-docker-secret` command-line argument of the `nginx-gateway` container. ### NGINX Plus Secret Place the JWT in a file called `license.jwt`. Create a Kubernetes Secret using the contents of the JWT file. ```shell kubectl create secret generic nplus-license --from-file license.jwt -n nginx-gateway ``` You can now delete the `license.jwt` file. If you need to update the JWT at any time, update the `license.jwt` field in the Secret using `kubectl edit` and apply the changes. If using a name other than the default `nplus-license`, provide the name of this Secret when installing NGINX Gateway Fabric: #### Helm Specify the Secret name using the `nginx.usage.secretName` helm value. #### Manifests Specify the Secret name in the `--usage-report-secret` command-line flag on the `nginx-gateway` container. **Note:** If you are reporting to the default licensing endpoint, then you can now proceed with [installing NGINX Gateway Fabric](/ngf/install/). Otherwise, follow the steps below to configure reporting to NGINX Instance Manager. ### Reporting to NGINX Instance Manager {#nim} If you are deploying NGINX Gateway Fabric in an environment where you need to report to NGINX Instance Manager instead of the default licensing endpoint, a few extra steps may be required. First, you must specify the endpoint of your NGINX Instance Manager: #### Helm Specify the endpoint using the `nginx.usage.endpoint` helm value. #### Manifests Specify the endpoint in the `--usage-report-endpoint` command-line flag on the `nginx-gateway` container. #### CA and Client certificate/key {#nim-cert} To configure a CA cert and/or client certificate and key, a few extra steps are needed. First, you need to create two Secrets in the `nginx-gateway` namespace. The CA must live under the key `ca.crt`: ```shell kubectl -n nginx-gateway create secret generic nim-ca --from-file ca.crt ``` The client cert and key must be added to a TLS Secret: ```shell kubectl -n nginx-gateway create secret tls nim-client --cert /path/to/cert --key /path/to/key ``` #### Helm Specify the CA Secret name using the `nginx.usage.caSecretName` helm value. Specify the client Secret name using the `nginx.usage.clientSSLSecretName` helm value. #### Manifests Specify the CA Secret name in the `--usage-report-ca-secret` command-line flag on the `nginx-gateway` container. Specify the client Secret name in the `--usage-report-client-ssl-secret` command-line flag on the `nginx-gateway` container.
**Note:** Once these Secrets are created and configuration options are set, you can now [install NGINX Gateway Fabric](/ngf/install/). ## Installation flags to configure usage reporting {#flags} When installing NGINX Gateway Fabric, the following flags can be specified to configure usage reporting to fit your needs: If using Helm, the `nginx.usage` values should be set as necessary: - `secretName` should be the name of the JWT Secret you created. By default this field is set to `nplus-license`. This field is required. - `endpoint` is the endpoint to send the telemetry data to. This is optional, and by default is `product.connect.nginx.com`. - `resolver` is the nameserver used to resolve the NGINX Plus usage reporting endpoint. This is optional and used with NGINX Instance Manager. - `skipVerify` disables client verification of the NGINX Plus usage reporting server certificate. - `caSecretName` is the name of the Secret containing the NGINX Instance Manager CA certificate. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). - `clientSSLSecretName` is the name of the Secret containing the client certificate and key for authenticating with NGINX Instance Manager. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). If using manifests, the following command-line options should be set as necessary on the `nginx-gateway` container: - `--usage-report-secret` should be the name of the JWT Secret you created. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). By default this field is set to `nplus-license`. - `--usage-report-endpoint` is the endpoint to send the telemetry data to. This is optional, and by default is `product.connect.nginx.com`. - `--usage-report-resolver` is the nameserver used to resolve the NGINX Plus usage reporting endpoint. This is optional and used with NGINX Instance Manager. - `--usage-report-skip-verify` disables client verification of the NGINX Plus usage reporting server certificate. - `--usage-report-ca-secret` is the name of the Secret containing the NGINX Instance Manager CA certificate. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). - `--usage-report-client-ssl-secret` is the name of the Secret containing the client certificate and key for authenticating with NGINX Instance Manager. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). ## What’s reported and how it’s protected {#telemetry} NGINX Plus reports the following data every hour by default: - **NGINX version and status**: The version of NGINX Plus running on the instance. - **Instance UUID**: A unique identifier for each NGINX Plus instance. - **Traffic data**: - **Bytes received from and sent to clients**: HTTP and stream traffic volume between clients and NGINX Plus. - **Bytes received from and sent to upstreams**: HTTP and stream traffic volume between NGINX Plus and upstream servers. - **Client connections**: The number of accepted client connections (HTTP and stream traffic). - **Requests handled**: The total number of HTTP requests processed. - **NGINX uptime**: The number of reloads and worker connections during uptime. - **Usage report timestamps**: Start and end times for each usage report. - **Kubernetes node details**: Information about Kubernetes nodes. ### Security and privacy of reported data All communication between your NGINX Plus instances, NGINX Instance Manager, and F5’s licensing endpoint (`product.connect.nginx.com`) is protected using **SSL/TLS** encryption. Only **operational metrics** are reported — no **personally identifiable information (PII)** or **sensitive customer data** is transmitted. ## Pull an image for local use To pull an image for local use, use this command: ```shell docker login private-registry.nginx.com --username= --password=none ``` Replace the contents of `` with the contents of the JWT token itself. You can then pull the image: ```shell docker pull private-registry.nginx.com/nginx-gateway-fabric/nginx-plus: ``` Once you have successfully pulled the image, you can tag it as needed, then push it to a different container registry. For a complete list of available NGINX Plus images, including UBI-based and WAF variants, see [Supported container images](/ngf/overview/technical-specifications.md#supported-container-images). ## Alternative installation options There are alternative ways to get an NGINX Plus image for NGINX Gateway Fabric: - [Build the Gateway Fabric image](/ngf/install/build-image.md) describes how to use the source code with an NGINX Plus subscription certificate and key to build an image. ## Next steps - [Deploy a Gateway for data plane instances](/ngf/install/deploy-data-plane.md) - [Routing traffic to applications](/ngf/traffic-management/basic-routing.md) -------------------------------------------------------------------------------- title: "Use Manifests to install NGINX Gateway Fabric (experimental) with NGINX Open Source" url: https://docs.nginx.com/nginx-gateway-fabric/install/manifests/open-source-experimental/index.md -------------------------------------------------------------------------------- This page describes how to use Manifests to install NGINX Gateway Fabric (experimental) with NGINX Open Source. It explains how to install the Gateway API resources and add authentication certificates, then deploy NGINX Gateway Fabric and its custom resource definitions. Using experimental NGINX Gateway Fabric versions allows to test API resources from upcoming releases as outlined by the [Milestone Roadmap](https://github.com/orgs/nginx/projects/10/views/5). By following these instructions, you will finish with a functional NGINX Gateway Fabric instance for your Kubernetes cluster. **Note:** To learn which Gateway API resources NGINX Gateway Fabric currently supports, view the [Gateway API Compatibility](/ngf/overview/gateway-api-compatibility.md) topic. ## Before you begin To complete this guide, you will need the following pre-requisites: - [A supported Kubernetes version](/ngf/overview/technical-specifications.md) - A functional Kubernetes cluster ## Install the Gateway API resources **Note:** If you have already installed Gateway API resources in your cluster, ensure they are a version [supported by NGINX Gateway Fabric](/ngf/overview/technical-specifications.md) To install API resources from the experimental channel, use `kubectl kustomize`: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl apply --server-side -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io/backendtlspolicies.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/xbackendtrafficpolicies.gateway.networking.x-k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/xlistenersets.gateway.networking.x-k8s.io serverside-applied ``` You should also create the _nginx-gateway_ namespace, which is used by the Manifest files by default: ```shell kubectl create namespace nginx-gateway ``` ## Install cert-manager Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ## Add certificates for secure authentication **Note:** These steps use a self-signed issuer, which should not be used in production environments. For production environments, you should use a real [CA issuer](https://cert-manager.io/docs/configuration/ca/). First, create a CA (certificate authority) issuer: ```yaml kubectl apply -f - <= v1.18.0, the default value changed from `Never` to `Always`. certificate.cert-manager.io/nginx-gateway-ca created issuer.cert-manager.io/nginx-gateway-issuer created ``` You will then need to create a server certificate for the NGINX Gateway Fabric control plane (server): **Note:** The default service name is _nginx-gateway_, and the namespace is _nginx-gateway_, so the `dnsNames` value should be `nginx-gateway.nginx-gateway.svc`. This value becomes the name of the NGINX Gateway Fabric control plane service. ```yaml {hl_lines=[13]} kubectl apply -f - <>gw: Request gw->>app: HTTPS request gw->>app: start TLS handshake app->>gw: request client certificate gw->>app: present client certificate from Secret: gateway-presents-this-cert-for-validation app->>app: validate client certificate using ca.crt in Secret: app-tls-secret app->>gw: present backend certificate gw->>gw: validate backend certificate using BackendTLSPolicy and ca.crt in Secret: backend-cert app->>gw: complete TLS handshake app-->>gw: HTTPS response gw-->>client: Response ``` ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Set up Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` Create a self-signed `ClusterIssuer`, a CA `Certificate`, and a CA-backed `ClusterIssuer`. cert-manager uses the resulting `local-ca-issuer` to sign certificates in any namespace: ```yaml kubectl apply -f - < 8443/TCP 9s ``` ## Configure routing rules First, create the Gateway resource with an HTTP listener: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. --- ## Send traffic without mutual TLS configured Using the external IP address and port for the NGINX Service, send traffic to the secure-app application. To show what happens before we configure backend TLS and have the Gateway present its client certificate for verification, send a request now and observe how the connection to the application fails with a bad request error. **Note:** If you have a DNS record allocated for `secure-app.example.com`, you can send the request directly to that hostname, without needing to resolve. ```shell curl --resolve secure-app.example.com:$GW_PORT:$GW_IP http://secure-app.example.com:$GW_PORT/ ``` ```text 400 The plain HTTP request was sent to HTTPS port

400 Bad Request

The plain HTTP request was sent to HTTPS port
nginx/1.29.2
``` We can see a status 400 Bad Request message from NGINX. --- ## Configure TLS for Gateway and Backend applications To create a Secret named `gateway-presents-this-cert-for-validation` signed by the local CA that Gateway presents to verify its identity, copy and paste the following command: ```yaml kubectl apply -f - < Annotations: API Version: gateway.networking.k8s.io/v1 Kind: BackendTLSPolicy Metadata: Creation Timestamp: 2025-11-13T23:28:36Z Generation: 1 Resource Version: 1288 UID: d7e3f026-afe3-44d1-aed5-c168e954b52f Spec: Target Refs: Group: Kind: Service Name: secure-app Validation: Ca Certificate Refs: Group: Kind: Secret Name: backend-cert Hostname: secure-app.example.com Status: Ancestors: Ancestor Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Namespace: default Conditions: Last Transition Time: 2025-11-13T23:28:37Z Message: All CACertificateRefs are resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs Last Transition Time: 2025-11-13T23:28:37Z Message: The Policy is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Controller Name: gateway.nginx.org/nginx-gateway-controller Events: ``` --- ## Send traffic with backend TLS configuration Now send traffic again: ```shell curl --resolve secure-app.example.com:$GW_PORT:$GW_IP http://secure-app.example.com:$GW_PORT/ ``` ```text hello from pod secure-app ``` To verify that the backend validated the gateway’s client certificate, inspect the logs of the `secure-app` pod and check the reported client subject: ```shell POD_NAME=$(kubectl get pod -l app=secure-app -o jsonpath='{.items[0].metadata.name}') kubectl logs "$POD_NAME" ``` ```text 10.244.0.145 ssl_client_verify=SUCCESS ssl_client_subject=CN=gateway ``` --- ## See also To learn more about configuring backend TLS termination using the Gateway API, see the following resources: - [Backend TLS Policy](https://gateway-api.sigs.k8s.io/api-types/backendtlspolicy/) - [Backend TLS Policy GEP](https://gateway-api.sigs.k8s.io/geps/gep-1897/) - [Gateway Backend TLS](https://gateway-api.sigs.k8s.io/reference/spec/#gatewaybackendtls) -------------------------------------------------------------------------------- title: "Technical specifications" url: https://docs.nginx.com/nginx-gateway-fabric/overview/technical-specifications/index.md description: NGINX Gateway Fabric technical specifications. -------------------------------------------------------------------------------- This page describes the technical specifications for NGINX Gateway Fabric. The information included covers version compatibility between NGINX Gateway Fabric and the Gateway API, as well as other NGINX products. ## NGINX Gateway Fabric versions The following table lists the software versions NGINX Gateway Fabric supports. Only the latest patch release for each minor version is shown. | NGINX Gateway Fabric | Gateway API | Kubernetes | NGINX OSS | NGINX Plus | NGINX Agent | F5 WAF for NGINX | |----------------------|-------------|------------|-----------|------------|-------------|------------------| | Edge | 1.6.1 | 1.32+ | 1.31.4 | R37.1 | v3.11.4 | 5.15.0 | | 2.7.0 | 1.6.1 | 1.32+ | 1.31.4 | R37.1 | v3.11.4 | 5.15.0 | | 2.6.8 | 1.5.1 | 1.31+ | 1.31.3 | R37.0 | v3.11.2 | 5.13.2 | | 2.5.1 | 1.5.1 | 1.31+ | 1.29.7 | R36 | v3.8.0 | --- | | 2.4.2 | 1.4.1 | 1.25+ | 1.29.5 | R36 | v3.7.1 | --- | | 2.3.0 | 1.4.1 | 1.25+ | 1.29.3 | R36 | v3.6.0 | --- | | 2.2.2 | 1.3.0 | 1.25+ | 1.29.2 | R35 | v3.6.0 | --- | | 2.1.4 | 1.3.0 | 1.25+ | 1.29.1 | R35 | v3.3.1 | --- | | 2.0.2 | 1.3.0 | 1.25+ | 1.28.0 | R34 | v3.0.1 | --- | | 1.6.2 | 1.2.1 | 1.25+ | 1.27.4 | R33 | --- | --- | | 1.5.1 | 1.2.0 | 1.25+ | 1.27.2 | R33 | --- | --- | | 1.4.0 | 1.1.0 | 1.25+ | 1.27.1 | R32 | --- | --- | | 1.3.0 | 1.1.0 | 1.25+ | 1.27.0 | R32 | --- | --- | | 1.2.0 | 1.0.0 | 1.23+ | 1.25.4 | R31 | --- | --- | **Note:** For version requirements to export F5 WAF for NGINX security events to NGINX Instance Manager, see [Export security logs to F5 NGINX Instance Manager](/ngf/waf-integration/policy-sources.md#export-security-logs-to-f5-nginx-instance-manager). ### OpenShift Compatibility The following table lists the OpenShift versions and Operator versions compatible with NGINX Gateway Fabric. | NGINX Gateway Fabric | Operator | Preferred Gateway API | Compatible Gateway API | OCP with Preferred GWAPI | Supported OCP Versions | |----------------------|----------|-----------------------|------------------------|--------------------------|------------------------| | 2.7.x | v1.5.x | v1.6.x | v1.2.1-v1.6.x | --- | 4.19 - 4.22 | | 2.6.x | v1.4.x | v1.5.x | v1.2.1-v1.5.x | --- | 4.19 - 4.21 | | 2.5.x | v1.3.x | v1.5.x | v1.2.1-v1.5.x | --- | 4.19 - 4.21 | | 2.4.x | v1.2.x | v1.4.x | v1.2.1-v1.4.x | 4.20 & 4.21 | 4.19 - 4.21 | | 2.2.x | v1.0.x | v1.3.0 | v1.2.1 | --- | 4.19 | NGINX Gateway Fabric is conformant with the Gateway API version installed on supported OCP versions. The "OCP with Preferred GWAPI" column shows which OCP versions ship with the preferred Gateway API version. On OCP versions with an older Gateway API installed, NGF remains fully conformant with that installed version, but features from newer Gateway API versions that NGF supports will be unavailable. ## Supported container images NGINX Gateway Fabric provides container images for the control plane and the NGINX data plane. All images are available for `amd64` and `arm64` architectures unless otherwise noted. ### Control plane images The control plane image contains the NGINX Gateway Fabric binary. | Name | Base image | Image | Architectures | |-----------------|------------------------|--------------------------------------------------------------|----------------| | Default image | `scratch` | `ghcr.io/nginx/nginx-gateway-fabric:` | amd64
arm64 | | UBI-based image | `redhat/ubi10-minimal` | `ghcr.io/nginx/nginx-gateway-fabric:-ubi` | amd64
arm64 | ### Data plane images with NGINX | Name | Base image | Image | Architectures | |-----------------|-----------------------|--------------------------------------------------------------------|----------------| | Default image | `alpine:3.24` | `ghcr.io/nginx/nginx-gateway-fabric/nginx:` | amd64
arm64 | | UBI-based image | `redhat/ubi10-minimal` | `ghcr.io/nginx/nginx-gateway-fabric/nginx:-ubi` | amd64
arm64 | ### Data plane images with NGINX Plus NGINX Plus images are available through the F5 Container registry `private-registry.nginx.com`. For setup instructions and authentication details, see [Install NGINX Gateway Fabric with NGINX Plus](/ngf/install/nginx-plus.md). | Name | Base image | Image | Architectures | |---------------------------------------|-----------------------|--------------------------------------------------------------------------------------------|----------------| | Default image | `alpine:3.24` | `private-registry.nginx.com/nginx-gateway-fabric/nginx-plus:` | amd64
arm64 | | UBI-based image | `redhat/ubi10-minimal` | `private-registry.nginx.com/nginx-gateway-fabric/nginx-plus:-ubi` | amd64
arm64 | | Default image with F5 WAF for NGINX | `alpine:3.24` | `private-registry.nginx.com/nginx-gateway-fabric/nginx-plus-f5waf:` | amd64 | | UBI-based image with F5 WAF for NGINX | `redhat/ubi10-minimal` | `private-registry.nginx.com/nginx-gateway-fabric/nginx-plus-f5waf:-ubi` | amd64 | ### WAF sidecar images When F5 WAF for NGINX is enabled, two additional sidecar containers are deployed alongside the NGINX container. These images are available from the F5 Container registry. | Name | Image | Architectures | |--------------------|---------------------------------------------------------------------------------|-------| | WAF Enforcer | `private-registry.nginx.com/nap/waf-enforcer:` | amd64 | | WAF Config Manager | `private-registry.nginx.com/nap/waf-config-mgr:` | amd64 | For more information on WAF integration, see [F5 WAF for NGINX overview](/ngf/waf-integration/overview.md). ### Custom images You can build custom NGINX Gateway Fabric images from source. For instructions, see [Build NGINX Gateway Fabric](/ngf/install/build-image.md). --- ## Gateway API compatibility The following tables summarizes which Gateway API resources NGINX Gateway Fabric supports and to which level. You can read more information by viewing the [Gateway API compatibility](/ngf/overview/gateway-api-compatibility.md) topic, or by selecting the resource name to go directly to the full details. **Note:** You can view the [Milestone Roadmap](https://github.com/orgs/nginx/projects/10/views/5) in the NGINX Gateway Fabric GitHub project to see what API resources will be supported in upcoming releases. | Resource | Core Support Level | Extended Support Level | Implementation-Specific Support Level | API Version | API Release Channel | |---------------------------------------|---------------------|------------------------|---------------------------------------|-------------|---------------------| | [GatewayClass](/ngf/overview/gateway-api-compatibility.md#gatewayclass) | Supported | Not supported | Supported | v1 | Standard | | [Gateway](/ngf/overview/gateway-api-compatibility.md#gateway) | Supported | Partially supported | Not supported | v1 | Standard | | [HTTPRoute](/ngf/overview/gateway-api-compatibility.md#httproute) | Supported | Partially supported | Not supported | v1 | Standard | | [GRPCRoute](/ngf/overview/gateway-api-compatibility.md#grpcroute) | Supported | Partially supported | Not supported | v1 | Standard | | [ReferenceGrant](/ngf/overview/gateway-api-compatibility.md#referencegrant) | Supported | N/A | Not supported | v1 | Standard | | [TLSRoute](/ngf/overview/gateway-api-compatibility.md#tlsroute) | Supported | Not supported | Not supported | v1 | Standard | | [TCPRoute](/ngf/overview/gateway-api-compatibility.md#tcproute) | Supported | Supported | Not supported | v1 | Standard | | [UDPRoute](/ngf/overview/gateway-api-compatibility.md#udproute) | Supported | Supported | Not supported | v1 | Standard | | [BackendTLSPolicy](/ngf/overview/gateway-api-compatibility.md#backendtlspolicy) | Partially supported | Supported | Partially supported | v1 | Standard | | [Custom policies](/ngf/overview/gateway-api-compatibility.md#custom-policies) | N/A | N/A | Supported | N/A | N/A | -------------------------------------------------------------------------------- title: "Configure basic authentication" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-security/basic-authentication/index.md -------------------------------------------------------------------------------- This page describes how to configure basic authentication in NGINX Gateway Fabric using the AuthenticationFilter custom resource definition (CRD). Authentication can be used to secure applications and APIs, ensuring only trusted and authorized users have access. By following these instructions, you will create two sample application endpoints. One will include basic authentication and the other will not, allowing you to review how each behaves. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Setup In this part of the document, we will set up several resources in your cluster to demonstrate usage of the AuthenticationFilter CRD. ## Deploy sample applications To deploy the `coffee` and `tea` applications, run the following YAML with `kubectl apply`: ```yaml kubectl apply -f - < ``` ## Create a user credentials secret and AuthenticationFilter Deploy a secret with user credentials, and the AuthenticationFilter by running the following YAML with `kubectl apply`: ```yaml kubectl apply -f - < ``` ## Deploy a HTTPRoute referencing the AuthenticationFilter Deploy a HTTPRoute resource which references the AuthenticationFilter using the `ExtensionRef` filter type. In this example, the filter is applied to the `/coffee` path: run the following YAML with `kubectl apply` ```yaml kubectl apply -f - < ``` ## Verify Basic Authentication **Note:** Your clients should be able to resolve the domain name "cafe.example.com" to the public IP of the NGINX Service. This guide simulates it using curl's `--resolve` option. Accessing `/coffee` with valid credentials: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -u user1:password1 ``` Response: ```text Server address: 10.244.0.7:8080 Server name: coffee-654ddf664b-nhhvr Date: 06/Jan/2026:15:20:15 +0000 URI: /coffee Request ID: 13a925b2514b62c45ea4a79800248d5c ``` Accessing `/coffee` without credentials: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` Response: ```text 401 Authorization Required

401 Authorization Required

nginx
``` Accessing `/coffee` with incorrect credentials: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -u user1:wrong ``` Response: ```text 401 Authorization Required

401 Authorization Required

nginx
``` Accessing `/tea` Since tea has no AuthenticationFilter attached, responses are processed normally: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` Response: ```text Server address: 10.244.0.10:8080 Server name: tea-75bc9f4b6d-ms2n8 Date: 06/Jan/2026:15:36:26 +0000 URI: /tea Request ID: c7eb0509303de1c160cb7e7d2ac1d99f ``` ## Troubleshooting - Ensure the HTTPRoute is Accepted and references the correct AuthenticationFilter name and group. - Confirm the secret key is named `auth`. - Ensure the secret referenced by the AuthenticationFilter is in the same namespace. ## Further reading - [Example deployment files for AuthenticationFilter](https://github.com/nginx/nginx-gateway-fabric/tree/main/examples/basic-authentication) - [NGINX HTTP Basic Auth Module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) -------------------------------------------------------------------------------- title: "Configure WAF settings" url: https://docs.nginx.com/nginx-gateway-fabric/waf-integration/configuration/index.md -------------------------------------------------------------------------------- This page covers operational configuration for F5 WAF for NGINX in NGINX Gateway Fabric: security logging, automatic policy updates, TLS and authentication, bundle integrity verification, cookie seed management, fetch failure handling, and WAF container settings. --- ## Configure security logging The `securityLogs` field on a `WAFPolicy` supports multiple log destinations. Each entry generates an `app_protect_security_log` directive in the NGINX configuration. ### Log destinations | Destination type | Description | |------------------|-----------------------------------------------------| | `stderr` | Write to standard error (appears in container logs) | | `file` | Write to a file path inside the NGINX container | | `syslog` | Send to a syslog server | ### Log source types For the `NIM`, `N1C`, and `HTTP` source types, each log entry must specify a `logSource` using one of: | Field | Description | |------------------|--------------------------------------| | `defaultProfile` | A built-in WAF log profile name | | `httpSource` | URL to a compiled log profile bundle | | `nimSource` | NGINX Instance Manager log profile configuration | | `n1cSource` | NGINX One Console log profile configuration | **Built-in log profiles:** `log_default`, `log_all`, `log_blocked`, `log_illegal`, `log_grpc_all`, `log_grpc_blocked`, `log_grpc_illegal` For the `PLM` source type, use `logRef.apLogConfRef` instead of `logSource` to reference an `APLogConf` custom resource by name and namespace. See [PLM (Policy Lifecycle Management)](/ngf/waf-integration/overview.md#plm-policy-lifecycle-management) for the full field comparison. ### Example: multiple log destinations ```yaml securityLogs: - destination: type: stderr logSource: defaultProfile: log_all - destination: type: file file: path: "/var/log/app_protect/security.log" logSource: httpSource: url: https://bundles.example.com/waf/custom-log-profile.tgz auth: secretRef: name: bundle-credentials validation: verifyChecksum: true - destination: type: syslog syslog: server: syslog-svc.default.svc.cluster.local:514 logSource: defaultProfile: log_blocked ``` --- ## Configure automatic policy updates (polling) Polling enables NGINX Gateway Fabric to detect and deploy updated policy bundles without modifying the `WAFPolicy` resource. This is useful when the same URL or policy name always resolves to the latest compiled bundle — for example, in a CI/CD workflow that overwrites the bundle file in place. Enable polling on a `policySource` or `logSource`: ```yaml policySource: nimSource: url: https://nim.example.com policyName: ngfBlocking auth: secretRef: name: nim-credentials polling: enabled: true interval: 10m # optional; defaults to 5m ``` **How polling works by source type:** - **NGINX Instance Manager and NGINX One Console**: NGINX Gateway Fabric first fetches only the bundle checksum or metadata. The full bundle is downloaded only if the checksum has changed, avoiding unnecessary traffic. - **HTTP**: NGINX Gateway Fabric sends a conditional GET using the stored `ETag` or `Last-Modified` header from the previous fetch. A `304 Not Modified` response skips the download entirely. **When not to enable polling:** If you pin a specific version — `policyUID` for NGINX Instance Manager, `policyVersionID` for NGINX One Console, or a version-specific URL for HTTP — the source always returns the same bundle. Every poll will detect "unchanged" and trigger no action. In that case, disable polling to avoid unnecessary network requests. **On poll failure:** If a poll attempt fails, the existing deployed bundle remains active. WAF protection is not interrupted. The error is recorded in the `WAFPolicy` status with reason `StaleBundleWarning`. --- ## Configure the WAF cookie seed When WAF is enabled, NGINX Gateway Fabric automatically sets the `app_protect_cookie_seed` NGINX directive to a stable value derived from the Gateway's UID. This ensures that WAF session cookies issued by one NGINX replica can be validated by any other replica in the same deployment. Without this, each replica generates its own random seed at startup, which causes cross-replica cookie validation failures. If you have pre-compiled the cookie seed into your WAF policy bundles using the [compiler global settings](/waf/configure/compiler.md), disable the automatic cookie seed in the NginxProxy CRD to avoid conflicting with the compiled-in value: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: waf-enabled-proxy spec: waf: enable: true disableCookieSeed: true ``` **Note:** Only set `disableCookieSeed: true` if your compiled bundles already contain a cookie seed. If neither the bundle nor NGINX Gateway Fabric provides a seed, each replica generates its own random value, which breaks WAF session cookie validation in multi-replica deployments. --- ## Configure TLS and authentication ### Custom CA certificate If your policy server uses a self-signed certificate or an internal CA, provide the CA certificate in a Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: custom-ca-secret type: Opaque data: ca.crt: ``` Reference it on the `policySource`: ```yaml policySource: httpSource: url: https://internal-server.example.com/policy.tgz tlsSecret: name: custom-ca-secret ``` The CA certificate is appended to the system CA pool. This is supported for all source types. **Note:** Do not set `insecureSkipVerify: true` in production environments. This disables TLS certificate verification and should be used only for local testing. ### Authentication methods NGINX Gateway Fabric infers the authentication method from the keys present in the Secret referenced by `auth.secretRef`: | Secret keys | Authentication method | |-------------------------|-----------------------------------------------------------------------| | `username` + `password` | HTTP Basic Auth | | `token` | Bearer Token (NGINX Instance Manager) or APIToken (NGINX One Console) | | (none) | No authentication header | Secrets must be in the same namespace as the `WAFPolicy`. --- ## Configure bundle integrity verification ### HTTP source Set `validation.verifyChecksum: true` to have NGINX Gateway Fabric fetch a companion `.sha256` file and compare its SHA-256 digest against the downloaded bundle. Any mismatch prevents the bundle from being deployed and sets `Programmed=False` with reason `IntegrityError`. Generate the companion file: ```shell sha256sum compiled-policy.tgz > compiled-policy.tgz.sha256 ``` Upload both files to the same location. NGINX Gateway Fabric appends `.sha256` to the bundle URL automatically. ### Known-checksum enforcement For any source type, you can pin an expected SHA-256 checksum to reject bundles that do not match: ```yaml policySource: nimSource: url: https://nim.example.com policyName: ngfBlocking validation: expectedChecksum: "abc1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcd" ``` The `expectedChecksum` must be a 64-character hexadecimal SHA-256 digest. **Note:** `verifyChecksum` and `expectedChecksum` are mutually exclusive. You can use one or the other on the same policy source, but not both. --- ## Policy fetch failure handling When a WAF policy bundle cannot be fetched — for example, due to a network error, authentication failure, or the bundle not yet being compiled — the behaviour of NGINX Gateway Fabric depends on whether this is the **first** fetch or a **subsequent** update. ### First-time fetch failure When the bundle has never been successfully fetched, you can control whether NGINX Gateway Fabric withholds the configuration push or allows traffic to flow unprotected while waiting for the bundle. This is configured using `waf.bundleFailOpen` on the `NginxProxy` resource: **Fail-closed (default, `bundleFailOpen: false`):** NGINX Gateway Fabric withholds the NGINX configuration push entirely until the bundle is available. No configuration changes — including unrelated route additions — are applied to the data plane while any pending bundle exists for the Gateway. The `app_protect_policy_file` directive is not emitted. The WAFPolicy status reflects the withheld push with `Programmed=False`. This is the safe default: the operator must resolve the bundle fetch failure before traffic is served. **Fail-open (`bundleFailOpen: true`):** NGINX Gateway Fabric pushes the NGINX configuration normally, but omits the pending WAFPolicy from the generated config — no `app_protect_policy_file` directive is emitted, so NGINX loads successfully without WAF protection. Traffic flows unprotected until the bundle becomes available, at which point NGINX Gateway Fabric includes the policy in the next configuration push. The WAFPolicy status continues to show `Programmed=False` while the bundle is pending, so operators are aware that WAF protection is not yet active. To enable fail-open, set `waf.bundleFailOpen: true` on the `NginxProxy`: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: waf-enabled-proxy spec: waf: enable: true bundleFailOpen: true ``` **Note:** `bundleFailOpen: true` means traffic is served without WAF protection if the initial bundle fetch fails. Use this only when availability is more critical than security posture during startup. Monitor the WAFPolicy status to confirm when the bundle becomes active. **Note:** The first-time fetch rules apply whenever NGINX Gateway Fabric starts without an already-fetched bundle. This includes upgrades, control plane pod restarts, and new Gateway deployments, because policy bundles are not persisted across pod restarts. With the default fail-closed setting, configuration pushes are withheld until all bundles have been re-fetched after each restart. Set `bundleFailOpen: true` if you need traffic to flow immediately after a restart and can accept a brief window without WAF protection. In both cases, the WAFPolicy status condition is set to `Programmed=False` with reason `Pending` until the bundle is successfully fetched. ### Subsequent fetch failures If a bundle fetch fails after a policy has already been successfully deployed — for example, during a polling cycle or after a WAFPolicy update — the **existing deployed bundle remains active**. WAF protection is not interrupted. The failure is reflected in the WAFPolicy status with `Programmed=True` and reason `StaleBundleWarning`, indicating that the previously fetched bundle is still in use: ```text - Type: Programmed Status: True Reason: StaleBundleWarning Message: policy bundle fetch failed; using previously fetched bundle: connection timeout ``` NGINX Gateway Fabric retries on the next reconciliation or poll cycle. No manual intervention is required unless the error persists. --- ## Configure WAF containers When WAF is enabled, NGINX Gateway Fabric deploys two sidecar containers — `waf-enforcer` and `waf-config-mgr` — alongside the main NGINX container. These settings are configured under `spec.kubernetes.deployment.wafContainers` (or `spec.kubernetes.daemonSet.wafContainers` for DaemonSet mode) in the NginxProxy resource. This follows the same infrastructure configuration pattern described in [Configure infrastructure-related settings](/ngf/how-to/data-plane-configuration.md#configure-infrastructure-related-settings). For the full list of configurable fields, see the `NginxProxy` spec in the [API reference](/ngf/reference/api.md). Each container (`enforcer` and `configManager`) supports the following fields: - **`image`**: Override the default image repository, tag, and pull policy. If not specified, NGINX Gateway Fabric uses the defaults from the F5 Container registry. For the default images, see [Supported container images](/ngf/overview/technical-specifications.md#supported-container-images). - **`resources`**: Set CPU and memory requests and limits. - **`volumeMounts`**: Add extra volume mounts. NGINX Gateway Fabric automatically configures the shared volumes required for communication between the NGINX, `waf-enforcer`, and `waf-config-mgr` containers. Additional mounts are appended to these defaults. The following example uses custom images from a private registry and sets resource requirements for both containers: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: waf-enabled-proxy spec: waf: enable: true kubernetes: deployment: wafContainers: enforcer: image: repository: registry.example.com/nap/waf-enforcer tag: "" resources: requests: cpu: 100m memory: 128Mi limits: cpu: "1" memory: 1Gi configManager: image: repository: registry.example.com/nap/waf-config-mgr tag: "" resources: requests: cpu: 50m memory: 64Mi limits: cpu: 500m memory: 256Mi ``` When installing with Helm, set the equivalent values under `nginx.wafContainers`: ```yaml # values.yaml nginx: config: waf: enable: true wafContainers: enforcer: image: repository: registry.example.com/nap/waf-enforcer tag: "" resources: requests: cpu: 100m memory: 128Mi configManager: image: repository: registry.example.com/nap/waf-config-mgr tag: "" resources: requests: cpu: 50m memory: 64Mi ``` **Note:** Image pull Secrets for private registries must be configured at install time using the `nginx.imagePullSecret` or `nginx.imagePullSecrets` Helm values (or the `--nginx-docker-secret` flag for manifest installs). The control plane copies these Secrets into any namespace where NGINX is deployed. For details, see [Install NGINX Gateway Fabric with NGINX Plus](/ngf/install/nginx-plus.md). --- ## See also - [F5 WAF for NGINX overview](/ngf/waf-integration/overview.md) - [Configure policy sources (NGINX Instance Manager and NGINX One Console)](/ngf/waf-integration/policy-sources.md) - [Configure infrastructure-related settings](/ngf/how-to/data-plane-configuration.md#configure-infrastructure-related-settings) - [Troubleshoot WAFPolicy status](/ngf/waf-integration/troubleshooting.md) - [Supported container images](/ngf/overview/technical-specifications.md#supported-container-images) - [WAFPolicy and NginxProxy API reference](/ngf/reference/api.md) - [Build and use the compiler tool](/waf/configure/compiler.md) -------------------------------------------------------------------------------- title: "Control plane configuration" url: https://docs.nginx.com/nginx-gateway-fabric/how-to/control-plane-configuration/index.md -------------------------------------------------------------------------------- Learn how to dynamically update the NGINX Gateway Fabric control plane configuration. ## Overview NGINX Gateway Fabric can dynamically update the control plane configuration without restarting. The control plane configuration is stored in the NginxGateway custom resource, created during the installation of NGINX Gateway Fabric. NginxGateway is deployed in the same namespace as the controller (Default: `nginx-gateway`). The resource's default name is based on your [installation method](/ngf/install/): - Helm: `-config` - Manifests: `nginx-gateway-config` The control plane only watches this single instance of the custom resource. If the resource is invalid to the OpenAPI schema, the Kubernetes API server will reject the changes. If the resource is deleted or deemed invalid by NGINX Gateway Fabric, a warning event is created in the `nginx-gateway` namespace, and the default values will be used by the control plane for its configuration. Additionally, the control plane updates the status of the resource (if it exists) to reflect whether it is valid or not. **For a full list of configuration options that can be set, see the `NginxGateway spec` in the [API reference](/ngf/reference/api.md).** --- ## Viewing and Updating the Configuration **Note:** For the following examples, the name `ngf-config` should be updated to the name of the resource created for your installation. To view the current configuration and its status: ```shell kubectl -n nginx-gateway describe nginxgateways ngf-config ``` ```text ... Status: Conditions: Last Transition Time: 2024-08-13T19:22:14Z Message: NginxGateway is valid Observed Generation: 1 Reason: Valid Status: True Type: Valid ``` To update the configuration: ```shell kubectl -n nginx-gateway edit nginxgateways ngf-config ``` This will open the configuration in your default editor. You can then update and save the configuration, which is applied automatically to the control plane. -------------------------------------------------------------------------------- title: "HTTP redirects and rewrites" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/redirects-and-rewrites/index.md -------------------------------------------------------------------------------- Learn how to redirect or rewrite your HTTP traffic using NGINX Gateway Fabric. ## Overview [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) filters can be used to configure HTTP redirects or rewrites. Redirects return HTTP 3XX responses to a client, instructing it to retrieve a different resource. Rewrites modify components of a client request (such as hostname and/or path) before proxying it upstream. In this guide, we will set up the coffee application to demonstrate path URL rewriting, and the tea and soda applications to showcase path-based request redirection. For an introduction to exposing your application, we recommend that you follow the [basic guide](/ngf/traffic-management/basic-routing.md) first. To see an example of a redirect using scheme and port, see the [HTTPS Termination](/ngf/traffic-management/https-termination.md) guide. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## HTTP rewrites and redirects examples We will configure a common gateway for the `URLRewrite` and `RequestRedirect` filter examples mentioned below. ### Deploy the Gateway resource for the applications The [Gateway](https://gateway-api.sigs.k8s.io/api-types/gateway/) resource is typically deployed by the [Cluster Operator](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/#roles-and-personas_1). This Gateway defines a single listener on port 80. Since no hostname is specified, this listener matches on all hostnames. To deploy the Gateway: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. ## URLRewrite example This examples demonstrates how to rewrite the traffic URL for a simple coffee application. An HTTPRoute resource is used to define two `URLRewrite` filters that will rewrite requests. You can verify the server responds with the rewritten URL. ### Deploy the coffee application Create the **coffee** application in Kubernetes by copying and pasting the following block into your terminal: ```yaml kubectl apply -f - < 80/TCP 40s ``` ### Configure a path rewrite The following HTTPRoute defines two filters that will rewrite requests such as the following: - `http://cafe.example.com/coffee` to `http://cafe.example.com/beans` - `http://cafe.example.com/coffee/flavors` to `http://cafe.example.com/beans` - `http://cafe.example.com/latte/prices` to `http://cafe.example.com/prices` To create the httproute resource, copy and paste the following into your terminal: ```yaml kubectl apply -f - < 80/TCP 89m service/tea ClusterIP 10.96.151.194 80/TCP 120m ``` ### Configure a path redirect In this section, we'll define two HTTPRoutes for the tea and soda applications to demonstrate different types of request redirection using the `RequestRedirect` filter. 1. The `tea-redirect` route uses the `ReplacePrefixMatch` type for path modification. This configuration matches the prefix of the original path and updates it to a new path, preserving the rest of the original URL structure. It will redirect request as follows: - `http://cafe.example.com/tea` to `http://cafe.example.com/organic` - `http://cafe.example.com/tea/origin` to `http://cafe.example.com/organic/origin` 2. The `soda-redirect` route uses the `ReplaceFullPath` type for path modification. This configuration updates the entire original path to the new location, effectively overwriting it. It will redirect request as follows: - `http://cafe.example.com/soda` to `http://cafe.example.com/flavors` - `http://cafe.example.com/soda/pepsi` to `http://cafe.example.com/flavors` To create the httproute resource, copy and paste the following into your terminal: ```yaml kubectl apply -f - < **Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. Use a regular NGINX Plus instance JWT. ## Create license and registry secrets First, create the _nginx-gateway_ namespace, which is used by the Manifest files by default: ```shell kubectl create namespace nginx-gateway ``` **Note:** The commands in the rest of this document should be run in the same directory as your **license.jwt** file. JWTs are sensitive information and should be stored securely. Delete them after use to prevent unauthorized access. Once you have obtained your license JWT, create a Kubernetes secret using `kubectl create`: ```shell kubectl create -n nginx-gateway secret generic nplus-license --from-file license.jwt ``` Then create another Kubernetes secret to allow interactions with the F5 registry: ```shell kubectl create -n nginx-gateway secret docker-registry nginx-plus-registry-secret \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat license.jwt) \ --docker-password=none ``` You can verify the creation of the secrets using `kubectl get`: ```shell kubectl get -n nginx-gateway secrets ``` #### Example output ```text NAME TYPE DATA AGE nginx-plus-registry-secret kubernetes.io/dockerconfigjson 1 8s nplus-license Opaque 1 21s ``` ## Install the Gateway API resources **Note:** If you have already installed Gateway API resources in your cluster, ensure they are a version [supported by NGINX Gateway Fabric](/ngf/overview/technical-specifications.md) To install API resources from the experimental channel, use `kubectl kustomize`: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl apply --server-side -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io/backendtlspolicies.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/xbackendtrafficpolicies.gateway.networking.x-k8s.io serverside-applied customresourcedefinition.apiextensions.k8s.io/xlistenersets.gateway.networking.x-k8s.io serverside-applied ``` ## Install cert-manager Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ## Add certificates for secure authentication **Note:** These steps use a self-signed issuer, which should not be used in production environments. For production environments, you should use a real [CA issuer](https://cert-manager.io/docs/configuration/ca/). First, create a CA (certificate authority) issuer: ```yaml kubectl apply -f - <= v1.18.0, the default value changed from `Never` to `Always`. certificate.cert-manager.io/nginx-gateway-ca created issuer.cert-manager.io/nginx-gateway-issuer created ``` You will then need to create a server certificate for the NGINX Gateway Fabric control plane (server): **Note:** The default service name is _nginx-gateway_, and the namespace is _nginx-gateway_, so the `dnsNames` value should be `nginx-gateway.nginx-gateway.svc`. This value becomes the name of the NGINX Gateway Fabric control plane service. ```yaml {hl_lines=[13]} kubectl apply -f - <_` ```yaml kubectl apply -f - < 53/UDP,53/TCP 10d ``` NGINX Gateway Fabric creates an `NginxProxy` resource during installation. Edit it to add the `dnsResolver` field: ```bash kubectl edit nginxproxies.gateway.nginx.org -n nginx-gateway ngf-proxy-config ``` ```yaml spec: dnsResolver: addresses: - type: IPAddress value: 10.96.0.10 ``` ### Create the Keycloak Secret This Secret holds the client secret and the CA certificate NGINX uses to verify Keycloak's TLS certificate on outbound connections. The CA certificate is extracted from the `keycloak-tls-cert` Secret that cert-manager created. ```shell kubectl create secret generic keycloak-secret \ --from-literal=client-secret=$CLIENT_SECRET \ --from-file=ca.crt=<(kubectl get secret keycloak-tls-cert -o jsonpath='{.data.ca\.crt}' | base64 -d) ``` ### Create the AuthenticationFilter The `AuthenticationFilter` defines how NGINX communicates with the IdP. The only required fields are `issuer`, `clientID`, and `clientSecretRef`. Everything else is optional. ```yaml kubectl apply -f - < ``` ### Deploy the HTTPRoutes Create an HTTPRoute with two rules. The `/coffee` rule attaches the `AuthenticationFilter`, whereas `/tea` is publicly accessible with no authentication required. ```yaml kubectl apply -f - < ``` --- ## Verify OIDC authentication For local testing, add the following entry to your `/etc/hosts` file so your browser can resolve `cafe.example.com` to the Gateway's public IP: ```text cafe.example.com ``` The steps below use a browser for OIDC since the flow involves redirects and cookies that curl cannot handle end-to-end. ### Accessing the protected `/coffee` route Open `https://cafe.example.com:$GW_PORT/coffee` in a browser. Because the route has an `AuthenticationFilter`, NGINX will: 1. Detect there is no valid session cookie. 1. Redirect your browser to the IdP's login page. 1. Log in with username `testuser` and password `testpassword`. 1. After you log in, redirect you back to NGINX with an authorization code. 1. Exchange the code for tokens in the background, set a session cookie, and forward you to the `coffee` backend. You will see a response from the `coffee` application only after a successful login. ### Accessing the unprotected `/tea` route Since `/tea` requires no authentication, you can access it directly with curl: ```shell curl -k --resolve cafe.example.com:$GW_PORT:$GW_IP https://cafe.example.com:$GW_PORT/tea ``` ```text Server address: 10.244.0.10:8080 Server name: tea-75bc9f4b6d-ms2n8 Date: 17/Mar/2026:10:01:00 +0000 URI: /tea Request ID: c7eb0509303de1c160cb7e7d2ac1d99f ``` The `tea` backend responds immediately with no authentication challenge because no `AuthenticationFilter` is attached to that rule. ## Optional configuration ### Session management By default, NGINX issues a session cookie named `NGX_OIDC_SESSION` with an 8-hour timeout that resets on each request to a protected resource. Use `session.cookieName` and `session.timeout` to override these values. ```yaml spec: type: OIDC oidc: session: cookieName: my-app-session timeout: 30m ``` ### Logout Use `logout.uri` to set the path a user visits to log out. When a request hits that path, NGINX clears the session and redirects to the IdP's logout endpoint. If `logout.postLogoutURI` is not set, NGINX returns a `200 OK` with the body "You have been logged out.". It can be set to a path to redirect the user there after logout. The path must be matched by an existing HTTPRoute rule. Set it to a full URL to redirect the user to an external page. Use `logout.frontChannelLogoutURI` if your IdP uses front-channel logout, where the IdP sends a logout request to a browser-visible URL to clear the NGINX session. The IdP must send `iss` and `sid` as query parameters. Set `logout.tokenHint` to `true` if your IdP requires the original ID token to be passed in the logout request. ```yaml spec: type: OIDC oidc: logout: uri: /logout postLogoutURI: /after_logout frontChannelLogoutURI: /frontchannel_logout tokenHint: true ``` ### PKCE PKCE (Proof Key for Code Exchange) prevents authorization code interception attacks. NGINX enables it automatically when the IdP requires the `S256` code challenge method. Set `pkce` explicitly if you need to force it on or off. ```yaml spec: type: OIDC oidc: pkce: true ``` ### Extra authentication arguments Use `extraAuthArgs` to append additional query parameters to the authorization request sent to the IdP. For example, `prompt: "login"` forces the IdP to show the login page on every request, and `max_age` sets the maximum time in seconds since the user last authenticated before re-authentication is required. ```yaml spec: type: OIDC oidc: extraAuthArgs: prompt: "login" max_age: "3600" ``` ### Custom redirect URI By default, NGINX Gateway Fabric registers the OIDC callback at `/oidc_callback__`. Use `redirectURI` to set a different path. If you provide a path-only value, NGINX creates a location block to handle the callback. If you provide a full URL, it is treated as an external handler and no location block is created. Register the same value in your IdP as an allowed redirect URI. ```yaml spec: type: OIDC oidc: redirectURI: /my_callback ``` ### Custom IdP metadata URL By default, NGINX fetches IdP metadata from `/.well-known/openid-configuration`. Use `configURL` if your IdP exposes metadata at a different path. ```yaml spec: type: OIDC oidc: configURL: "https://keycloak.example.com/realms/my-realm/.well-known/openid-configuration" ``` ### Certificate Revocation List (CRL) A CRL is a list of certificate serial numbers that a CA has revoked before expiry. When NGINX connects to the IdP over TLS, it checks the IdP's certificate against the CRL and rejects the connection if the certificate has been revoked. You are responsible for keeping the CRL up to date. A stale CRL may not catch recently revoked certificates. The `crlSecretRef` and `caCertificateRefs` fields are separate so you can rotate the CRL independently, though both keys can live in the same Secret. ```yaml spec: type: OIDC oidc: crlSecretRef: name: oidc-crl ``` --- ## Troubleshooting ### AuthenticationFilter is not accepted - Confirm the filter's `type` is `OIDC` and the `oidc` block is present. - Check that the Secrets referenced by `clientSecretRef`, `caCertificateRefs`, and `crlSecretRef` exist in the same namespace as the filter and contain the expected keys (`client-secret`, `ca.crt`, `ca.crl`). ### HTTPRoute is not accepted or reports `ResolvedRefs=False` - Verify the `extensionRef` in the HTTPRoute matches the `AuthenticationFilter` name and namespace exactly. - Confirm the route's `parentRefs` points to an HTTPS listener. OIDC filters are rejected on non-HTTPS listeners. ### Browser is stuck in a redirect loop - Confirm the `redirectURI` registered in the IdP exactly matches the path NGINX is using (default: `/oidc_callback__`). - Ensure the Gateway's TLS certificate is valid for the hostname the browser is using. ## Further reading - [NGINX OIDC module reference](https://nginx.org/en/docs/http/ngx_http_oidc_module.html) - [How OpenID Connect works](https://openid.net/developers/how-connect-works/) - [Single Sign-On with OpenID Connect and Identity Providers](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-oidc) -------------------------------------------------------------------------------- title: "Data plane configuration" url: https://docs.nginx.com/nginx-gateway-fabric/how-to/data-plane-configuration/index.md -------------------------------------------------------------------------------- Learn how to dynamically update the NGINX Gateway Fabric global data plane configuration. ## Overview NGINX Gateway Fabric can dynamically update the global data plane configuration without restarting. The data plane configuration contains configuration for NGINX that is not available using the standard Gateway API resources. This includes options such as configuring an OpenTelemetry collector, disabling HTTP/2, changing the IP family, and setting the NGINX error log level. It also contains options for [updating the NGINX Deployment and Service config](#configure-infrastructure-related-settings), either through native fields or patches. The data plane configuration is stored in the `NginxProxy` custom resource, which is a namespace-scoped resource that can be attached to a GatewayClass or Gateway. When attached to a GatewayClass, the fields in the NginxProxy affect all Gateways that belong to the GatewayClass. When attached to a Gateway, the fields in the NginxProxy only affect the Gateway. If a GatewayClass and its Gateway both specify an NginxProxy, the GatewayClass NginxProxy provides defaults that can be overridden by the Gateway NginxProxy. See the [Merging Semantics](#merging-semantics) section for more detail. **For a full list of configuration options that can be set, see the `NginxProxy spec` in the [API reference](/ngf/reference/api.md).** --- ## Merging Semantics NginxProxy resources are merged when a GatewayClass and a Gateway reference different NginxProxy resources. For fields that are bools, integers, and strings: - If a field on the Gateway's NginxProxy is unspecified (`nil`), the Gateway __inherits__ the value of the field in the GatewayClass's NginxProxy. - If a field on the Gateway's NginxProxy is specified, its value __overrides__ the value of the field in the GatewayClass's NginxProxy. For array fields: - If the array on the Gateway's NginxProxy is unspecified (`nil`), the Gateway __inherits__ the entire array in the GatewayClass's NginxProxy. - If the array on the Gateway's NginxProxy is empty, it __overrides__ the entire array in the GatewayClass's NginxProxy, effectively unsetting the field. - If the array on the Gateway's NginxProxy is specified and not empty, it __overrides__ the entire array in the GatewayClass's NginxProxy. ### Merging Examples This section contains examples of how NginxProxy resources are merged when they are attached to both a Gateway and its GatewayClass. #### Disable HTTP/2 for a Gateway A GatewayClass references the following NginxProxy which explicitly allows HTTP/2 traffic and sets the IPFamily to ipv4: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: gateway-class-enable-http2 namespace: default spec: ipFamily: "ipv4" disableHTTP: false ``` To disable HTTP/2 traffic for a particular Gateway, reference the following NginxProxy in the Gateway's spec: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: gateway-disable-http namespace: default spec: disableHTTP: true ``` These NginxProxy resources are merged and the following settings are applied to the Gateway: ```yaml ipFamily: "ipv4" disableHTTP: true ``` #### Change Telemetry configuration for a Gateway A GatewayClass references the following NginxProxy which configures telemetry: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: gateway-class-telemetry namespace: default spec: telemetry: exporter: endpoint: "my.telemetry.collector:9000" interval: "60s" batchSize: 20 serviceName: "my-company" spanAttributes: - key: "company-key" value: "company-value" ``` To change the telemetry configuration for a particular Gateway, reference the following NginxProxy in the Gateway's spec: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: gateway-telemetry-service-name namespace: default spec: telemetry: exporter: batchSize: 50 batchCount: 5 serviceName: "my-app" spanAttributes: - key: "app-key" value: "app-value" ``` These NginxProxy resources are merged and the following settings are applied to the Gateway: ```yaml telemetry: exporter: endpoint: "my.telemetry.collector:9000" interval: "60s" batchSize: 50 batchCount: 5 serviceName: "my-app" spanAttributes: - key: "app-key" value: "app-value" ``` #### Disable Tracing for a Gateway A GatewayClass references the following NginxProxy which configures telemetry: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: gateway-class-telemetry namespace: default spec: telemetry: exporter: endpoint: "my.telemetry.collector:9000" interval: "60s" serviceName: "my-company" ``` To disable tracing for a particular Gateway, reference the following NginxProxy in the Gateway's spec: ```yaml apiVersion: gateway.nginx.org/v1alpha2 kind: NginxProxy metadata: name: gateway-disable-tracing namespace: default spec: telemetry: disabledFeatures: - DisableTracing ``` These NginxProxy resources are merged and the following settings are applied to the Gateway: ```yaml telemetry: exporter: endpoint: "my.telemetry.collector:9000" interval: "60s" serviceName: "my-app" disabledFeatures: - DisableTracing ``` --- ## Configuring the GatewayClass NginxProxy on install By default, an `NginxProxy` resource is created in the same namespace where NGINX Gateway Fabric is installed, attached to the GatewayClass. You can set configuration options in the `nginx` Helm value section, and the resource will be created and attached using the set values. You can also [manually create and attach](#manually-creating-nginxProxies) specific `NginxProxy` resources to target different Gateways. When installed using the Helm chart, the NginxProxy resource is named `-proxy-config` and is created in the release Namespace. **For a full list of configuration options that can be set, see the `NginxProxy spec` in the [API reference](/ngf/reference/api.md).** **Note:** Some global configuration also requires an [associated policy](/ngf/overview/custom-policies.md) to fully enable a feature (such as [tracing](/ngf/monitoring/tracing.md), for example). --- ## Manually Creating NginxProxies The following command creates a basic `NginxProxy` configuration in the `default` namespace that sets the IP family to `ipv4` instead of the default value of `dual`: ```yaml kubectl apply -f - < ``` This will open your default editor, allowing you to add the following to the `spec`: ```yaml infrastructure: parametersRef: group: gateway.nginx.org kind: NginxProxy name: ngf-proxy-config ``` **Note:** The `NginxProxy` resource must reside in the same namespace as the Gateway it is attached to. After updating, you can check the status of the Gateway to see if the configuration is valid: ```shell kubectl describe gateway ``` ```text ... Status: Conditions: ... Message: parametersRef resource is resolved Observed Generation: 1 Reason: ResolvedRefs Status: True Type: ResolvedRefs ``` If everything is valid, the `ResolvedRefs` condition should be `True`. Otherwise, you will see an `InvalidParameters` condition in the status. --- ## Configure the data plane log level You can use the `NginxProxy` resource to dynamically configure the log level. The following command creates a basic `NginxProxy` configuration that sets the log level to `warn` instead of the default value of `info`: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. ## RequestHeaderModifier example This examples demonstrates how to configure traffic routing for a simple echo server. A HTTPRoute resource is used to route traffic to the headers application, using the `RequestHeaderModifier` filter to modify headers in the request. You can then verify that the server responds with the modified request headers. ### Deploy the Headers application Begin by deploying the example application `headers`. It is a simple application that returns the request headers which will be modified later. ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/http-request-header-filter/headers.yaml ``` This will create the headers Service and a Deployment with one Pod. Run the following command to verify the resources were created: ```shell kubectl get pods,svc ``` ```text pod/headers-545698447b-z52kj 1/1 Running 0 23s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/headers ClusterIP 10.96.26.161 80/TCP 23s ``` ### Configure the HTTPRoute with RequestHeaderModifier filter Create a HTTPRoute that exposes the header application outside the cluster using the listener created in the previous section. Use the following command: ```yaml kubectl apply -f - < 80/TCP 95s ``` ### Configure the basic HTTPRoute Create a HTTPRoute that exposes the header application outside the cluster using the listener created in the previous section. You can do this with the following command: ```yaml kubectl apply -f - <` to inspect status conditions. This page documents all condition types, reasons, and common troubleshooting steps. --- ## Condition reference ### Accepted | Status | Reason | Meaning | |---------|--------------------|-----------------------------------------------------------------------------| | `True` | `Accepted` | Policy is valid and targets a known resource | | `False` | `Invalid` | Policy spec fails validation (for example, wrong source field for the type) | | `False` | `TargetNotFound` | The targeted Gateway or Route does not exist | | `False` | `Conflicted` | Another WAFPolicy already targets this resource at the same level | | `False` | `NginxProxyNotSet` | WAF is not enabled in the referenced NginxProxy | ### ResolvedRefs | Status | Reason | Meaning | |---------|--------------------|--------------------------------------------------------------------------| | `True` | `ResolvedRefs` | All referenced Secrets, `APPolicy`, and `APLogConf` resources resolved successfully | | `False` | `InvalidRef` | A referenced Secret was not found or is missing expected keys; or a referenced `APPolicy`/`APLogConf` doesn't exist | | `False` | `RefNotPermitted` | A referenced `APPolicy` or `APLogConf` is in a different namespace and no `ReferenceGrant` permits the reference | ### Programmed | Status | Reason | Meaning | |---------|----------------------|----------------------------------------------------------------------------------| | `True` | `Programmed` | Bundle fetched and deployed to the data plane | | `True` | `BundleUpdated` | A poll cycle detected a changed bundle and deployed it | | `True` | `StaleBundleWarning` | A poll cycle failed; previously deployed bundle remains active | | `False` | `FetchError` | Bundle could not be fetched (network error, HTTP error, auth failure, timeout) | | `False` | `IntegrityError` | Bundle checksum verification failed | | `False` | `Pending` | Bundle has never been fetched; configuration withheld or WAF omitted (fail-open) | --- ## Common issues ### `FetchError` with HTTP 403 The credentials Secret is either missing, contains the wrong keys, or the credentials are invalid. Verify the Secret exists in the same namespace as the `WAFPolicy` and that the keys match the authentication method (`username`/`password` for Basic Auth, `token` for Bearer/APIToken). ### `FetchError` with HTTP 404 on NGINX Instance Manager or NGINX One Console The referenced policy was not found or has not been compiled yet. For NGINX Instance Manager, verify that compilation succeeded in the NGINX Instance Manager console before creating the `WAFPolicy`. For NGINX One Console, NGINX Gateway Fabric triggers compilation if no bundle exists, and a 404 after initial setup may indicate the policy was deleted in NGINX One Console. ### `Pending` The bundle has never been successfully fetched. If `bundleFailOpen` is `false` (the default), the NGINX configuration push is withheld for this Gateway. If `bundleFailOpen` is `true`, traffic flows without WAF protection. Check the `Programmed` condition message for the last fetch error. Common causes include network connectivity issues, incorrect URLs, or authentication failures. Verify the policy source URL and credentials Secret. ### `IntegrityError` The downloaded bundle does not match the expected checksum. For HTTP source, ensure the `.sha256` file matches the bundle file. For `expectedChecksum`, verify the digest matches the bundle you intend to deploy. ### Policy not applied to a route The route does not show a `gateway.nginx.org/WAFPolicyAffected` condition. Verify that: - The `WAFPolicy` `targetRefs` field matches the Gateway or Route name and namespace. - The Gateway has `waf.enable: true` in its referenced `NginxProxy`. - The `WAFPolicy` `Accepted` condition is `True`. ### WAF sidecars not starting Verify that the `waf-enforcer` and `waf-config-mgr` container images are accessible from your cluster, and that any required `imagePullSecrets` are configured in the `NginxProxy` Kubernetes spec. ### Duplicate policy name error If two `WAFPolicy` resources in the same Gateway reference different compiled bundles that were compiled under the **same policy name**, the WAF engine rejects the configuration with an error like: ```text "error_message": "Duplicate policy name found: " ``` This occurs because the WAF engine uses the logical policy name embedded inside the compiled bundle — not the Kubernetes resource name or bundle filename — to identify policies. When the same logical name appears more than once in a single NGINX configuration, the configuration test fails and the update is rolled back. **How to identify the problem:** Check the NGINX Gateway Fabric controller logs for a configuration error containing `Duplicate policy name found`: ```shell kubectl logs -n nginx-gateway deploy/nginx-gateway -c nginx-gateway | grep "Duplicate policy name" ``` **Resolution:** Each `WAFPolicy` attached to a Gateway must reference a compiled bundle with a unique logical policy name. This is the `name` field set inside the policy definition JSON at compile time, not the `WAFPolicy` resource name or the bundle filename. To resolve the conflict, choose one of the following approaches: - **Recompile with a distinct name**: Update the policy definition to use a unique `name` field for each policy, then recompile and republish the bundle. - **Pin a single version per Gateway**: If the intent is to apply the same policy everywhere, use a single gateway-level `WAFPolicy` instead of multiple route-level policies referencing different versions of the same named policy. - **Check for overlapping WAFPolicies**: Run `kubectl get wafpolicies -A` and confirm that no two policies targeting the same Gateway reference bundles compiled from definitions with the same logical policy name. --- ### Security events aren't reaching NGINX Instance Manager F5 WAF for NGINX generates security events, but they don't appear in the NGINX Instance Manager Security Monitoring dashboard, even though the `WAFPolicy` resource shows `Programmed`. **How to identify the problem:** Check whether the event reached NGINX Agent inside the pod: ```shell kubectl exec -n -c nginx -- \ tail -100 /var/log/nginx-agent/opentelemetry-collector-agent.log ``` If the event isn't in this log, F5 WAF for NGINX isn't reaching NGINX Agent. If the event is in the log but not in NGINX Instance Manager, the export from NGINX Agent is failing. **Resolution:** - **Event missing from the NGINX Agent log:** Confirm the `WAFPolicy` `securityLogs.destination.syslog.server` field is set to exactly `localhost:1514`. Any other value prevents the event from reaching NGINX Agent, which listens on `127.0.0.1:1514` inside the `nginx` container. - **Event in the log but export fails:** Check the log for `Unauthenticated` errors. A JWT authentication failure between NGINX Agent and NGINX Instance Manager causes this error. This is the same NGINX Plus subscription JWT used to create the NGINX Plus Secret in [Connect NGINX Gateway Fabric to NGINX Instance Manager](/nim/connect-kubernetes/connect-ngf.md). If the JWT has expired or was revoked, download a new one from [MyF5](https://my.f5.com/manage/s/) and repeat the steps to recreate the Secret. NGINX Gateway Fabric doesn't pick up a rotated Secret automatically. Restart the Gateway pod after recreating it. - **Export succeeds but NGINX Instance Manager shows nothing:** Confirm NGINX Instance Manager's embedded OpenTelemetry collector is running and reachable on port `4317`. See [Troubleshooting](/nim/security-monitoring/troubleshooting.md) for the NGINX Instance Manager–side checks. --- ## See also - [F5 WAF for NGINX overview](/ngf/waf-integration/overview.md) - [Configure WAF settings](/ngf/waf-integration/configuration.md) - [WAFPolicy and NginxProxy API reference](/ngf/reference/api.md) -------------------------------------------------------------------------------- title: "Use Manifests to uninstall NGINX Gateway Fabric" url: https://docs.nginx.com/nginx-gateway-fabric/install/manifests/uninstall/index.md -------------------------------------------------------------------------------- This page describes how to uninstall NGINX Gateway Fabric in a Manifest-based deployment, its custom resource definitions, and its Gateway API resources. By following these instrucions, you will remove NGINX Gateway Fabric and all Gateway resources from a Kubernetes cluster. ## Remove NGINX Gateway Fabric To remove NGINX Gateway Fabric run: ```shell kubectl delete namespace nginx-gateway kubectl delete clusterrole nginx-gateway kubectl delete clusterrolebinding nginx-gateway ``` ## Delete the custom resource definitions To remove the NGINX Gateway Fabric custom resource definitions, run: ```shell kubectl delete -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/deploy/crds.yaml ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io "authenticationfilters.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "clientsettingspolicies.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "nginxgateways.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "nginxproxies.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "observabilitypolicies.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "proxysettingspolicies.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "ratelimitpolicies.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "snippetsfilters.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "snippetspolicies.gateway.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "upstreamsettingspolicies.gateway.nginx.org" deleted ``` ## Remove the Gateway API resources To uninstall the Gateway API resources, run the command based on your deployment type: #### Default ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v" | kubectl delete -f - ``` #### Experimental ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl delete -f - ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io "backendtlspolicies.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "gateways.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "httproutes.gateway.networking.k8s.io" deleted customresourcedefinition.apiextensions.k8s.io "referencegrants.gateway.networking.k8s.io" deleted ``` -------------------------------------------------------------------------------- title: "Configure external authentication" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-security/external-authentication/index.md -------------------------------------------------------------------------------- This guide describes how to configure external authentication in NGINX Gateway Fabric using the `ExternalAuth` filter on an HTTPRoute. External authentication delegates the authorization decision for each request to an external service. NGINX issues a subrequest to that service before proxying the original request, and forwards the request only if the service responds with a 2xx status. Following these instructions to create two sample applications and compare the behavior of each: - `coffee` endpoint: Protected by an `ExternalAuth` filter. - `tea` endpoint: Exposed without any external authentication filter. ## Overview The `ExternalAuth` filter is declared in the `filters` list of an HTTPRoute rule. When NGINX processes a request that matches the rule, it first sends a subrequest to the backend referenced by the filter. Based on the status returned by that backend, NGINX either forwards the original request to the route's `backendRefs` or returns the error status to the client. Each route rule supports only one `ExternalAuth` filter. If your authentication flow requires multiple checks, consolidate them into a single authentication service that performs all the necessary validations. The filter translates to NGINX's [ngx_http_auth_request_module](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html) directives: - [`auth_request`](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html#auth_request) — Sends a subrequest to the specified URI and grants or denies access based on the response status. - [`auth_request_set`](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html#auth_request_set) — Captures a value from the authentication response and stores it in a variable for use in the main request. ## Note on Gateway API Experimental Features **important:** ExternalAuth is a Gateway API resource from the experimental release channel. To use Gateway API experimental resources, the Gateway API resources from the experimental channel must be installed before deploying NGINX Gateway Fabric. Additionally, NGINX Gateway Fabric must have experimental features enabled. **Note:** As noted in the [Gateway API documentation](https://gateway-api.sigs.k8s.io/guides/#install-experimental-channel), future releases of the Gateway API can include breaking changes to experimental resources and fields. To install the Gateway API resources from the experimental channel, run the following: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl apply -f - ``` **Note:** If you plan to use the `edge` version of NGINX Gateway Fabric, you can replace the version in `ref` with `main`, for example `ref=main`. To enable experimental features on NGINX Gateway Fabric: Using Helm: Set `nginxGateway.gwAPIExperimentalFeatures.enable` to true. An example can be found in the [Installation with Helm](/ngf/install/helm.md#custom-installation-options) guide. Using Kubernetes manifests: Add the `--gateway-api-experimental-features` command-line flag to the deployment manifest args. An example can be found in the [Installation with Kubernetes manifests](/ngf/install/manifests.md#3-deploy-nginx-gateway-fabric) guide. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric with experimental features enabled. ## Deploy sample applications Run the following `kubectl apply` command to create the `coffee` and `tea` deployments and services: ```yaml kubectl apply -f - < ``` ## Deploy the external authentication server This sample authentication service is an NGINX deployment that checks the `X-Api-Key` request header. If the header value is `my-custom-secret`, the server responds with `200 OK`; otherwise it responds with `401 Unauthorized`. ```yaml kubectl apply -f - <<'EOF' apiVersion: v1 kind: ConfigMap metadata: name: ext-auth-config data: default.conf: | server { listen 8080; location / { if ($http_x_api_key != "my-custom-secret") { return 401 "unauthorized"; } return 200 "ok"; } } --- apiVersion: apps/v1 kind: Deployment metadata: name: ext-auth-server spec: replicas: 1 selector: matchLabels: app: ext-auth-server template: metadata: labels: app: ext-auth-server spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 8080 volumeMounts: - name: config mountPath: /etc/nginx/conf.d volumes: - name: config configMap: name: ext-auth-config --- apiVersion: v1 kind: Service metadata: name: ext-auth-server spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: ext-auth-server EOF ``` ## Configure routing with the ExternalAuth filter Run the following `kubectl apply` command to create an HTTPRoute for `coffee` and `tea` applications. The `coffee` route uses an `ExternalAuth` filter to require authentication, while the `tea` route is exposed without one: ```yaml kubectl apply -f - < 401 Authorization Required

401 Authorization Required

nginx
``` Access `/coffee` with a valid API key: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -H "X-Api-Key: my-custom-secret" ``` ```text Server address: 10.244.0.151:8080 Server name: coffee-654ddf664b-l9ml5 Date: 16/Apr/2026:20:14:28 +0000 URI: /coffee Request ID: 217931bc5fe27254d1821cec91e1f2d8 ``` The `X-Api-Key` header is listed in `allowedHeaders` so that it reaches the authentication server, which responds `200 OK`. NGINX then proxies the request to the `coffee` backend. Access `/tea`, which has no `ExternalAuth` filter and responds normally: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` ```text Server address: 10.244.0.149:8080 Server name: tea-75bc9f4b6d-q5wg5 Date: 16/Apr/2026:20:14:41 +0000 URI: /tea Request ID: d27f6ef4edc2f1e09bb455824ac67a07 ``` ### Exceed the body size limit Because `forwardBody.maxSize: 1024` is applied as `client_max_body_size` on the `/coffee` location, any client request with a body larger than 1024 bytes is rejected with `413 Request Entity Too Large` before the authorization subrequest runs. Send a 1100-byte body to demonstrate this: ```shell BODY=$(head -c 1100 /dev/zero | tr '\0' 'x') curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -X POST -H "X-Api-Key: my-custom-secret" -d "$BODY" ``` ```text 413 Request Entity Too Large

413 Request Entity Too Large

nginx
``` `client_max_body_size` can also be set on a route through a [ClientSettingsPolicy](/ngf/traffic-management/client-settings.md) via its `body.maxSize` field. If a ClientSettingsPolicy with `body.maxSize` is attached to the same HTTPRoute as an `ExternalAuth` filter that sets `forwardBody.maxSize`, the HTTPRoute is marked invalid with reason `InvalidFilter`. ## Troubleshooting - If the HTTPRoute is not accepted, run `kubectl describe httproute coffee` and check the `Status` conditions for validation errors. - If every request returns `401`, confirm that the authentication server is reachable from the NGINX pod and that the `backendRef` name, namespace, and port are correct. - If a required request header cannot reach the authentication server, confirm it is listed in `http.allowedHeaders`. - If a response header from the authentication server cannot reach the backend, confirm it is listed in `http.allowedResponseHeaders`. - If a request is rejected with `413 Request Entity Too Large`, raise `forwardBody.maxSize` to accommodate the client body. - If the HTTPRoute reports `ResolvedRefs: False` with an `InvalidFilter` reason mentioning `body.maxSize`, remove either the `ExternalAuth` filter's `forwardBody.maxSize` or the ClientSettingsPolicy's `body.maxSize` as they both cannot be set on the same route. ## Further reading - [NGINX HTTP auth request module](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html) - [Gateway API HTTPExternalAuthFilter specification](https://gateway-api.sigs.k8s.io/reference/spec/#httpexternalauthfilter) -------------------------------------------------------------------------------- title: "Configure JSON Web Token (JWT) authentication" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-security/jwt-authentication/index.md -------------------------------------------------------------------------------- This guide describes how to configure JSON Web Token (JWT) authentication in NGINX Gateway Fabric using the AuthenticationFilter custom resource definition (CRD). JWT authentication secures applications and APIs by validating JSON Web Tokens in incoming requests. Only requests with valid JWTs are allowed access. By following these instructions, you will create two sample application endpoints: one with JWT authentication and one without, so you can see how each behaves. **Note:** JWT authentication requires NGINX Plus. ## Overview JWT authentication in NGINX Gateway Fabric validates JSON Web Tokens using JSON Web Key Sets (JWKS). The JWKS contains the public keys used to verify JWT signatures. When a request arrives with a JWT in the `Authorization` header, NGINX Plus validates the token against the configured JWKS before forwarding the request to your application. NGINX Gateway Fabric supports two JWKS source types, set using the `source` field on the `AuthenticationFilter`: - **File** — JWKS is stored locally in a Kubernetes Secret. Use this when you manage your own keys or want to avoid external dependencies. - **Remote** — NGINX Plus fetches JWKS from an HTTPS endpoint at runtime. Use this when your identity provider (for example, Keycloak or Auth0) exposes a JWKS URI. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric with NGINX Plus. ## Common setup The following steps are required for both file-based and remote JWT authentication. ### Deploy sample applications To deploy the `coffee` and `tea` applications, run the following YAML with `kubectl apply`: ```yaml kubectl apply -f - < ``` --- ## File-based JWT authentication Use file-based JWT authentication when your JWKS is stored in a Kubernetes Secret. NGINX Plus loads the key material directly from the Secret at startup and after each reload. ### Generate a JWKS For testing purposes, the following example shows a simple JWKS with a single RSA key. In production, use properly generated keys from your identity provider or key management system. ```json { "keys": [ { "kty": "RSA", "kid": "test-key", "use": "sig", "n": "0vx7agoebGcQSuuPiLJXZptN9nndrQmbXEps2aiAFbWhM78LhWx4cbbfAAtVT86zwu1RK7aPFFxuhDR1L6tSoc_BJECPebWKRXjBZCiFV4n3oknjhMstn64tZ_2W-5JsGY4Hc5n9yBXArwl93lqt7_RN5w6Cf0h4QyQ5v-65YGjQR0_FDW2QvzqY368QQMicAtaSqzs8KJZgnYb9c7d0zgdAZHzu6qMQvRL5hajrn1n91CbOpbISD08qNLyrdkt-bFTWhAI4vMQFh6WeZu0fM4lFd2NcRwr3XPksINHaQ-G_xBniIqbw0Ls1jF44-csFCur-kEgU8awapJzKnqDKgw", "e": "AQAB" } ] } ``` **Note:** This example JWKS is for demonstration only. In production, use keys from your identity provider or key management system. ### Create a JWKS Secret and AuthenticationFilter Deploy a Secret containing your JWKS and the AuthenticationFilter by running these `kubectl` commands: ``` kubectl create secret generic jwks-secret --from-file=auth=jwks.json ``` ```yaml kubectl apply -f - < ``` ### Deploy an HTTPRoute referencing the AuthenticationFilter Deploy an HTTPRoute that references the AuthenticationFilter using the `ExtensionRef` filter type. In this example, the filter is applied to the `/coffee` path only. Run the following YAML with `kubectl apply`: ```yaml kubectl apply -f - < ``` ### Verify file-based JWT authentication **Note:** Your clients should be able to resolve "cafe.example.com" to the public IP of the NGINX Service. This guide simulates that using curl's `--resolve` option. To test the authentication, you need a JWT signed with the private key that corresponds to the public key in your JWKS. You can use [jwt.io](https://jwt.io) or other JWT tools to generate one. Store it in a shell variable: ```shell JWT_TOKEN="" ``` Access `/coffee` with a valid JWT: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -H "Authorization: Bearer $JWT_TOKEN" ``` ```text Server address: 10.244.0.7:8080 Server name: coffee-654ddf664b-nhhvr Date: 10/Mar/2026:15:20:15 +0000 URI: /coffee Request ID: 13a925b2514b62c45ea4a79800248d5c ``` Access `/coffee` without a JWT: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` ```text 401 Authorization Required

401 Authorization Required

nginx
``` Access `/coffee` with an invalid JWT: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -H "Authorization: Bearer invalid.jwt.token" ``` ```text 401 Authorization Required

401 Authorization Required

nginx
``` Access `/tea`, which has no AuthenticationFilter and responds normally: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` ```text Server address: 10.244.0.10:8080 Server name: tea-75bc9f4b6d-ms2n8 Date: 10/Mar/2026:15:36:26 +0000 URI: /tea Request ID: c7eb0509303de1c160cb7e7d2ac1d99f ``` --- ## Remote JWT authentication Use remote JWT authentication when your identity provider (IdP) exposes a JWKS endpoint. NGINX Plus fetches the JWKS from the URI at runtime using an internal subrequest, so keys are always up to date without requiring a Secret or NGINX reload. ### Install cert-manager Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` ### Generate certificates The following steps use `cert-manager` to issue a local Certificate Authority (CA) and sign certificates for both Keycloak and NGINX. `cert-manager` creates the required Kubernetes Secrets directly so no manual secret creation is needed for TLS. Create a self-signed `ClusterIssuer` to bootstrap the CA, then issue the CA certificate and create a second `ClusterIssuer` backed by it: ```yaml kubectl apply -f - < ``` ### Deploy an HTTPRoute referencing the remote AuthenticationFilter Deploy an HTTPRoute that applies the remote AuthenticationFilter to `/coffee`. Run the following YAML with `kubectl apply`: ```yaml kubectl apply -f - < ``` ### Obtain a JWT Expose Keycloak with port-forward: ```shell kubectl port-forward svc/keycloak 8443:8443 ``` Set your `JWT_TOKEN` environment variable by calling the `tokens` endpoint in keycloak: ```shell export JWT_TOKEN=$(curl -s -k -X POST https://localhost:8443/realms/nginx-gateway/protocol/openid-connect/token \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "username=testuser" \ -d "password=testpassword" \ -d "grant_type=password" \ -d "client_id=cafe-app" | jq -r '.access_token') ``` ### Verify remote JWT authentication **Note:** Your clients should be able to resolve "cafe.example.com" to the public IP of the NGINX Service. This guide simulates that using curl's `--resolve` option. Access `/coffee` with a valid JWT: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -H "Authorization: Bearer $JWT_TOKEN" ``` ```text Server address: 10.244.0.7:8080 Server name: coffee-654ddf664b-nhhvr Date: 10/Mar/2026:15:20:15 +0000 URI: /coffee Request ID: 13a925b2514b62c45ea4a79800248d5c ``` Access `/coffee` without a JWT: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` ```text 401 Authorization Required

401 Authorization Required

nginx
``` Access `/tea`, which has no AuthenticationFilter and responds normally: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` ```text Server address: 10.244.0.10:8080 Server name: tea-75bc9f4b6d-ms2n8 Date: 10/Mar/2026:15:36:26 +0000 URI: /tea Request ID: c7eb0509303de1c160cb7e7d2ac1d99f ``` --- ## Troubleshooting - Ensure NGINX Gateway Fabric is deployed with NGINX Plus. JWT authentication is not supported in the open source version. - Ensure the HTTPRoute is accepted and references the correct AuthenticationFilter name and group. - For file-based JWT: confirm the Secret key is named `auth` and contains valid JWKS JSON. The Secret must be in the same namespace as the AuthenticationFilter. - For remote JWT: confirm the `uri` uses the `https://` scheme and the endpoint is reachable from the NGINX Plus pod. - For remote JWT with a custom CA: confirm the Secret key is named `ca.crt` and contains a valid PEM certificate. The Secret must be in the same namespace as the AuthenticationFilter. - Verify your JWT includes the `kid` (key ID) claim that matches one of the keys in your JWKS. - Check that the JWT is not expired by verifying the `exp` claim. - Ensure the JWT signature algorithm (typically RS256) matches the key type in your JWKS. ## Further reading - [AuthenticationFilter API reference](/ngf/reference/api.md#gateway.nginx.org/v1alpha1.AuthenticationFilter) - [NGINX JWT Authentication Module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html) - [JWT.io - JWT debugger and generator](https://jwt.io) - [RFC 7519 - JSON Web Token (JWT)](https://datatracker.ietf.org/doc/html/rfc7519) - [RFC 7517 - JSON Web Key (JWK)](https://datatracker.ietf.org/doc/html/rfc7517) -------------------------------------------------------------------------------- title: "Configure TLS routing with TLSRoute" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/tlsroute/index.md -------------------------------------------------------------------------------- Learn how to configure TLS routing with [TLSRoute](https://gateway-api.sigs.k8s.io/reference/spec/#tlsroute) using NGINX Gateway Fabric. ## Overview TLSRoute supports two TLS modes: - **Passthrough**: The Gateway reads the SNI and forwards encrypted TCP traffic to the backend. The backend holds and terminates TLS with its own certificate. Use this mode when the backend needs its own certificate, or when you can't expose the private key to the gateway. - **Terminate**: The Gateway holds the certificate, terminates TLS, and forwards plain TCP to the backend. Use this mode when the backend shouldn't handle TLS, or when it serves a non-HTTP TCP protocol. **note:** You can add an HTTPS listener on the same port that terminates TLS connections, as long as the hostname doesn't overlap with the TLS listener hostname. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. Set up cert-manager and a local CA for both examples: Install cert-manager onto the cluster using Helm with Gateway API features enabled. - Add the Helm repository. ```shell helm repo add jetstack https://charts.jetstack.io helm repo update ``` - Install cert-manager, and enable the GatewayAPI feature gate: ```shell helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \ --set config.kind="ControllerConfiguration" \ --set config.enableGatewayAPI=true \ --set crds.enabled=true ``` Create a self-signed `ClusterIssuer`, a CA `Certificate`, and a CA-backed `ClusterIssuer`. cert-manager uses the resulting `local-ca-issuer` to sign certificates in any namespace: ```yaml kubectl apply -f - < 8443/TCP 12s ``` Create a Gateway with a TLS listener in passthrough mode. Copy and paste this into your terminal: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the Gateway will forward for. Create a TLSRoute that attaches to the Gateway and routes requests to `app.example.com` to the `secure-app` Service: ```yaml kubectl apply -f - < GET / HTTP/1.1 > Host: app.example.com:8443 > User-Agent: curl/8.7.1 > Accept: */* > * Request completely sent off < HTTP/1.1 200 OK < Server: nginx/1.29.1 < Date: Wed, 06 May 2026 21:25:18 GMT < Content-Type: text/plain < Content-Length: 42 < Connection: keep-alive < hello from pod secure-app-59bbd475b-phgsv ``` Note that the server certificate used to terminate the TLS connection has the subject common name of `app.example.com`. This is the server certificate that the `secure-app` is configured with and shows that the TLS connection was terminated by the `secure-app`, not NGINX Gateway Fabric. ## TLS terminate In terminate mode, NGINX Gateway Fabric holds the TLS certificate, terminates the TLS connection, and forwards plain TCP traffic to the backend. The backend doesn't need a certificate or TLS configuration. Use TLS terminate mode when: - Your backend serves a non-HTTP TCP protocol, such as a database or custom binary protocol. - You want to centralize certificate management at the gateway rather than on each backend. **note:** If your backend serves HTTP traffic and you need HTTP-level routing — such as path matching or header manipulation — use an HTTPS listener with an HTTPRoute instead. See [Configure HTTPS termination](/ngf/traffic-management/https-termination.md). ### Set up Create a `Certificate` for `app.example.com`. cert-manager creates the `gateway-tls-secret` Secret, which the Gateway uses to terminate TLS: ```yaml kubectl apply -f - < 80/TCP 10s ``` Create a Gateway with a TLS listener in terminate mode. Copy and paste this into your terminal: ```yaml kubectl apply -f - < ``` **note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the Gateway will forward for. Create a TLSRoute that attaches to the Gateway and routes requests to `app.example.com` to the `app` Service: ```yaml kubectl apply -f - < ``` From the information obtained with `kubectl describe` you can see the default settings for the provisioned NGINX Deployment and Service. Under `Spec.Kubernetes` you can see a few things: - The NGINX container image settings - How many NGINX Deployment replicas are specified - The type of Service and external traffic policy **Note:** Depending on installation configuration, the default NginxProxy settings may be slightly different from what is shown in the example. For more information on NginxProxy and its configurable fields, see the [API reference](/ngf/reference/api.md). Modify the NginxProxy resource to change the type of Service. Use `kubectl edit` to modify the default NginxProxy and insert the following under `spec.kubernetes.service`: ```yaml type: NodePort ``` After saving the changes, use `kubectl get` on the service, and you should see the service type has changed to `NodePort`. ```shell kubectl get service cafe-nginx ``` ```text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cafe-nginx NodePort 10.96.172.204 80:32615/TCP 3h5m ``` ### Set annotations and labels on provisioned resources While the majority of configuration will happen on the NginxProxy resource, that is not always the case. Uniquely, if you want to set any annotations or labels on the NGINX Deployment or Service, you need to set those annotations on the Gateway which provisioned them. You can use `kubectl edit` on the Gateway and add the following to the `spec`: ```yaml infrastructure: annotations: annotationKey: annotationValue labels: labelKey: labelValue ``` After saving the changes, check the Service and NGINX deployment with `kubectl describe`. ```shell kubectl describe deployment cafe ``` ```text Name: cafe-nginx Namespace: default CreationTimestamp: Mon, 05 May 2025 16:49:33 -0700 ... Pod Template: Labels: app.kubernetes.io/instance=ngf app.kubernetes.io/managed-by=ngf-nginx app.kubernetes.io/name=cafe-nginx gateway.networking.k8s.io/gateway-name=cafe labelKey=labelValue Annotations: annotationKey: annotationValue prometheus.io/port: 9113 prometheus.io/scrape: true ... ``` ```shell kubectl describe service cafe-nginx ``` ```text Name: cafe-nginx Namespace: default Labels: app.kubernetes.io/instance=ngf app.kubernetes.io/managed-by=ngf-nginx app.kubernetes.io/name=cafe-nginx gateway.networking.k8s.io/gateway-name=cafe labelKey=labelValue Annotations: annotationKey: annotationValue ``` ## See also For more guides on routing traffic to applications and more information on Data Plane configuration, check out the following resources: - [Routing traffic to applications](/ngf/traffic-management/basic-routing.md) - [Application routes using HTTP matching conditions](/ngf/traffic-management/advanced-routing.md) - [Data plane configuration](/ngf/how-to/data-plane-configuration.md) - [API reference](/ngf/reference/api.md) -------------------------------------------------------------------------------- title: "Product telemetry" url: https://docs.nginx.com/nginx-gateway-fabric/overview/product-telemetry/index.md -------------------------------------------------------------------------------- Learn why, what and how NGINX Gateway Fabric collects telemetry. ## Overview NGINX Gateway Fabric collects telemetry by default, which allows its developers to understand what features are most popular with its user base. This data is used to triage development work, prioritizing features and functionality that will positively impact the most people. Telemetry data is collected once every 24 hours and sent to a service managed by F5 over HTTPS. Personally identifiable information (PII) is **not** collected. The list of data points that are collected can be seen below. **If you would prefer to not have data collected, you can [opt-out](#opt-out) when installing NGINX Gateway Fabric.** ## Collected data - **Kubernetes:** - **Platform:** the Kubernetes platform that NGINX Gateway Fabric is running on - **Version:** the Kubernetes version that NGINX Gateway Fabric is running on. - **Architecture:** the architecture that the Kubernetes environment is running on. - **Cluster UID:** the UID of the `kube-system` Namespace in the cluster that NGINX Gateway Fabric is running in. - **Cluster Node Count:** the number of Nodes in the cluster. - **Version:** the version of the NGINX Gateway Fabric Deployment. - **Deployment UID:** the UID of the NGINX Gateway Fabric Deployment. - **Image Build Source:** whether the image was built by GitHub or locally (values are `gha`, `local`, or `unknown`). The source repository of the images is **not** collected. - **Build OS:** the base operating system the image was built on (values are currently `alpine` or `ubi`). - **Deployment Flags:** a list of NGINX Gateway Fabric Deployment flags that are specified by a user. The actual values of non-boolean flags are **not** collected; we only record that they are either `true` or `false` for boolean flags and `default` or `user-defined` for the rest. - **Count of Resources:** the total count of resources related to NGINX Gateway Fabric. This includes supported Gateway API CRDs (such as `Gateways` and `HTTPRoutes`) and NGINX Gateway Fabric specific CRDs (such as `ClientSettingsPolicies` and `WAFPolicies`). The data within these resources is **not** collected. - **Snippets Info:** a list of directive-context strings from applied `SnippetFilters` and `SnippetsPolicies`, and a total count per strings. The actual value of any NGINX directive is **not** collected. - **Control Plane Pod Count:** the count of NGINX Gateway Fabric Pods. - **Data Plane Pod Count:** the count of NGINX data plane Pods. - **NGINX One Console Connection Info:** indicates whether the connection to the NGINX One Console is enabled. This data is used to identify the following information: - The flavors of Kubernetes environments that are most popular among our users. - The number of unique NGINX Gateway Fabric installations. - The scale of NGINX Gateway Fabric Deployments. - The scale of Gateway API resources. - The used features of NGINX Gateway Fabric. - The cluster is connected to NGINX One Console. Our goal is to publicly discuss data trends to drive roadmap discussions in our [Community Meeting](https://github.com/nginx/nginx-gateway-fabric/discussions/1472). ## Opt out You can disable product telemetry when installing NGINX Gateway Fabric using an option dependent on your installation method: ### Helm Set the `nginxGateway.productTelemetry.enable=false` flag either in the `values.yaml` file or when installing: ```shell helm install ... --set nginxGateway.productTelemetry.enable=false ``` ### Manifests Add the `--product-telemetry-disable` flag to the `nginx-gateway` container in your Deployment manifest. -------------------------------------------------------------------------------- title: "Troubleshooting" url: https://docs.nginx.com/nginx-gateway-fabric/troubleshooting/index.md -------------------------------------------------------------------------------- This topic describes possible issues when using NGINX Gateway Fabric and general troubleshooting techniques. When possible, suggested workarounds are provided. ### General troubleshooting When investigating a problem or requesting help, there are important data points that can be collected to help understand what issues may exist. #### Resource status To check the status of a resource, use `kubectl describe`. This example checks the status of the `coffee` HTTPRoute, which has an error: ```shell kubectl describe httproutes.gateway.networking.k8s.io coffee -n nginx-gateway ``` ```text ... Status: Parents: Conditions: Last Transition Time: 2024-05-31T17:20:51Z Message: The route is accepted Observed Generation: 4 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2024-05-31T17:20:51Z Message: spec.rules[0].backendRefs[0].name: Not found: "bad-backend" Observed Generation: 4 Reason: BackendNotFound Status: False Type: ResolvedRefs Controller Name: gateway.nginx.org/nginx-gateway-controller Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Namespace: default Section Name: http ``` If a resource has errors relating to its configuration or relationship to other resources, they can likely be read in the status. The `ObservedGeneration` in the status should match the `ObservedGeneration` of the resource. Otherwise, this could mean that the resource hasn't been processed yet or that the status failed to update. If no `Status` is written on the resource, further debug by checking if the referenced resources exist and belong to NGINX Gateway Fabric. #### Events Events created by NGINX Gateway Fabric or other Kubernetes components could indicate system or configuration issues. To see events: ```shell kubectl get events -n nginx-gateway ``` For example, a warning event when the NginxGateway configuration CRD is deleted: ```text kubectl -n nginx-gateway get event LAST SEEN TYPE REASON OBJECT MESSAGE 5s Warning ResourceDeleted nginxgateway/ngf-config NginxGateway configuration was deleted; using defaults ``` #### Get shell access to NGINX container Getting shell access to containers allows developers and operators to view the environment of a running container, see its logs or diagnose any problems. To get shell access to the NGINX container, use `kubectl exec`: ```shell kubectl exec -it -n -- /bin/sh ``` #### Logs Logs from the NGINX Gateway Fabric control plane and data plane can contain information that isn't available to status or events. These can include errors in processing or passing traffic. **Note:** You can see logs for a crashed or killed container by adding the `-p` flag to the `kubectl logs` commands below. 1. Container Logs To see logs for the control plane container: ```shell kubectl -n nginx-gateway logs -c nginx-gateway ``` To see logs for the data plane container: ```shell kubectl -n logs -c nginx ``` 1. Error Logs For the _nginx-gateway_ container, you can `grep` the logs for the word `error`: ```shell kubectl -n nginx-gateway logs -c nginx-gateway | grep error ``` For the _nginx_ container you can `grep` for various [error](https://nginx.org/en/docs/ngx_core_module.html#error_log) logs. For example, to search for all logs logged at the `emerg` level: ```shell kubectl -n logs -c nginx | grep emerg ``` For example, if a variable is too long, NGINX may display such an error message: ```text kubectl logs -n dev-env gateway-nginx-bb8598998-jwk2m -c nginx | grep emerg 2024/06/13 20:04:17 [emerg] 27#27: too long parameter, probably missing terminating """ character in /etc/nginx/conf.d/http.conf:78 ``` 1. Access Logs NGINX access logs record all requests processed by the NGINX server. These logs provide detailed information about each request, which can be useful for troubleshooting and analyzing web traffic. Access logs can be viewed with the above method of using `kubectl logs`, or by viewing the access log file directly. To do that, get shell access to your NGINX container using these [steps](#get-shell-access-to-nginx-container). The access logs are located in the file `/var/log/nginx/access.log` in the NGINX container. 1. Modify Log Levels To modify log levels for the control plane in NGINX Gateway Fabric, edit the `NginxGateway` configuration. This can be done either before or after deploying NGINX Gateway Fabric. Refer to this [guide](https://docs.nginx.com/nginx-gateway-fabric/how-to/control-plane-configuration/) to do so. To check error logs, modify the log level to `error` to view error logs. Similarly, change the log level to `debug` and `grep` for the word `debug` to view debug logs. #### Understanding the generated NGINX configuration Understanding the NGINX configuration is key for fixing issues because it shows how NGINX handles requests. This helps tweak settings to make sure NGINX behaves the way you want it to for your application. To see your current configuration, you can open a shell in the _nginx_ container by following these [steps](#get-shell-access-to-nginx-container) and run `nginx -T`. To understand the usage of NGINX directives in the configuration file, consult this list of [NGINX directives](https://nginx.org/en/docs/dirindex.html). In this section, we will see how the configuration gets updated as we configure different Services, Deployments and HTTPRoutes with NGINX Gateway Fabric. In the configuration file, you'll often find several server blocks, each assigned to specific ports and server names. NGINX selects the appropriate server for a request and evaluates the URI against the location directives within that block. When only a Gateway resource is defined, but no Services or HTTPRoutes are configured, NGINX generates a basic configuration. This includes a default server listening on the ports specified in the Gateway listeners, handling all incoming requests. Additionally, there are blocks to manage errors with status codes 500 or 503. This is a default `server` block listening on port 80: ```text server { listen 80 default_server; default_type text/html; return 404; } ``` Once an HTTPRoute with path matches and rules are defined, nginx.conf is updated accordingly to determine which location block will manage incoming requests. To demonstrate how `nginx.conf` is changed, create some resources: 1. A Gateway with single listener with the hostname `*.example.com` on port 80. 2. A simple `coffee` application. 3. An HTTPRoute that exposes the `coffee` application outside the cluster using the listener created in step 1. The path and rule matches create different location blocks in `nginx.conf` to route requests as needed. For example, this `coffee` route matches requests with path `/coffee` and type `prefix`. Examine how the `nginx.conf` is modified: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: coffee spec: parentRefs: - name: gateway sectionName: http hostnames: - "cafe.example.com" rules: - matches: - path: type: PathPrefix value: /coffee backendRefs: - name: coffee port: 80 ``` The modified `nginx.conf`: ```shell server { listen 80 default_server; default_type text/html; return 404; } server { listen 80; server_name cafe.example.com; location /coffee/ { proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_http_version 1.1; proxy_pass http://default_coffee_80$request_uri; } location = /coffee { proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_http_version 1.1; proxy_pass http://default_coffee_80$request_uri; } location / { return 404 ""; } } upstream default_coffee_80 { random two least_conn; zone default_coffee_80 512k; server 10.244.0.13:8080; } ``` Key information to note is: 1. A new `server` block is created with the hostname of the HTTPRoute. When a request is sent to this hostname, it will be handled by this `server` block. 1. Within the `server` block, three new `location` blocks are added for _coffee_, each with distinct prefix and exact paths. Requests directed to the _coffee_ application with a path prefix `/coffee/hello` will be managed by the first location block, while those with an exact path `/coffee` will be handled by the second location block. Any other requests not recognized by the server block for this hostname will default to the third location block, returning a 404 Not Found status. 1. Each `location` block has headers and directives that configure the NGINX proxy to forward requests to the `/coffee` path correctly, preserving important client information and ensuring compatibility with the upstream server. 1. The `upstream` block in the given NGINX configuration defines a group of backend servers and configures how NGINX should load balance requests among them. Review the behaviour when a curl request is sent to the `coffee` application: Matches location /coffee/ block ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee/hello Handling connection for 8080 Server address: 10.244.0.13:8080 Server name: coffee-56b44d4c55-hwpkp Date: 13/Jun/2024:22:51:52 +0000 URI: /coffee/hello Request ID: 21fc2baad77337065e7cf2cd57e04383 ``` Matches location = /coffee block ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee Handling connection for 8080 Server address: 10.244.0.13:8080 Server name: coffee-56b44d4c55-hwpkp Date: 13/Jun/2024:22:51:40 +0000 URI: /coffee Request ID: 4d8d719e95063303e290ad74ecd7339f ``` Matches location / block ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/ Handling connection for 8080 404 Not Found

404 Not Found

nginx/1.25.4
``` **Note:** The configuration may change in future releases. This configuration is valid for version 1.3. #### Metrics for troubleshooting Metrics can be useful to identify performance bottlenecks and pinpoint areas of high resource consumption within NGINX Gateway Fabric. To set up metrics collection, refer to the [Prometheus Metrics guide](/ngf/monitoring/prometheus.md). The metrics dashboard will help you understand problems with the way NGINX Gateway Fabric is set up or potential issues that could show up with time. #### Access the NGINX Plus Dashboard If you have NGINX Gateway Fabric installed with NGINX Plus, you can access the NGINX Plus dashboard at `http://localhost:8080/dashboard.html`. Verify that the port number (for example, `8080`) matches the port number you have port-forwarded to your NGINX Gateway Fabric Pod. For further details, see the [dashboard guide](monitoring/dashboard.md) ### Common errors | Problem Area | Symptom | Troubleshooting Method | Common Cause | |------------------------------|----------------------------------------|---------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------| | Startup | NGINX Gateway Fabric fails to start. | Check logs for _nginx_ and _nginx-gateway_ containers. | Readiness probe failed. | | Resources not configured | Status missing on resources. | Check referenced resources. | Referenced resources do not belong to NGINX Gateway Fabric. | | NGINX errors | Reload failures on NGINX | Fix permissions for control plane. | Security context not configured. | | NGINX Plus errors | Failure to start; traffic interruptions | Set up the [NGINX Plus JWT](/ngf/install/nginx-plus.md) | License is not configured or has expired. | | Client Settings | Request entity too large error | Adjust client settings. Refer to [Client Settings Policy](/ngf/traffic-management/client-settings.md) | Payload is greater than the [`client_max_body_size`](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) value.| ##### NGINX fails to reload NGINX reload errors can occur for various reasons, including syntax errors in configuration files, permission issues, and more. To determine if NGINX has failed to reload, check logs for your _nginx-gateway_ and _nginx_ containers. You will see the following error in the _nginx-gateway_ logs: `failed to reload NGINX:`, followed by the reason for the failure. Similarly, error logs in _nginx_ container start with `emerg`. For example, `2024/06/12 14:25:11 [emerg] 12345#0: open() "/var/run/nginx.pid" failed (13: Permission denied)` shows a critical error, such as a permission problem preventing NGINX from accessing necessary files. To debug why your reload has failed, start with verifying the syntax of your configuration files by opening a shell in the NGINX container following these [steps](#get-shell-access-to-nginx-container) and running `nginx -T`. If there are errors in your configuration file, the reload will fail and specify the reason for it. ##### NGINX Gateway Fabric Pod is not running or ready To understand why the NGINX Gateway Fabric Pod has not started running or is not ready, check the state of the Pod to get detailed information about the current status and events happening in the Pod. To do this, use `kubectl describe`: ```shell kubectl describe pod -n nginx-gateway ``` The Pod description includes details about the image name, tags, current status, and environment variables. Verify that these details match your setup and cross-check with the events to ensure everything is functioning as expected. For example, the Pod below has the nginx-gateway container that is running and the events reflect the same. ```text Containers: nginx-gateway: Container ID: containerd://492f380d5919ae2cdca0e009e7a7d5bf4092f8e1910f52d8951d58b73f125646 Image: nginx-gateway-fabric:latest Image ID: sha256:c034f1e5bde0490b1f2441e0e9b0bcfce5f2e259bb6210c55d4d67f808a74ecb Ports: 8443/TCP, 9113/TCP, 8081/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Args: controller --gateway-ctlr-name=gateway.nginx.org/nginx-gateway-controller --gatewayclass=nginx --config=my-release-config --service=my-release-nginx-gateway-fabric --agent-tls-secret=agent-tls --metrics-port=9113 --health-port=8081 --leader-election-lock-name=my-release-nginx-gateway-fabric-leader-election State: Running Started: Thu, 24 Apr 2025 10:57:16 -0700 Ready: True Restart Count: 0 Readiness: http-get http://:health/readyz delay=3s timeout=1s period=1s #success=1 #failure=3 Environment: POD_NAMESPACE: nginx-gateway (v1:metadata.namespace) POD_NAME: my-release-nginx-gateway-fabric-b99bd5cdd-qzp5q (v1:metadata.name) POD_UID: (v1:metadata.uid) INSTANCE_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) IMAGE_NAME: nginx-gateway-fabric:latest Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5dg45 (ro) /var/run/secrets/ngf from nginx-agent-tls (rw) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 22s default-scheduler Successfully assigned nginx-gateway/my-release-nginx-gateway-fabric-b99bd5cdd-qzp5q to kind-control-plane Normal Pulled 20s kubelet Container image "nginx-gateway-fabric:latest" already present on machine Normal Created 20s kubelet Created container: nginx-gateway Normal Started 20s kubelet Started container nginx-gateway ``` ##### NGINX Pod is not running or ready To understand why the NGINX Pod has not started running or is not ready, check the state of the Pod to get detailed information about the current status and events happening in the Pod. To do this, use `kubectl describe`: ```shell kubectl describe pod -n ``` The Pod description includes details about the image name, tags, current status, and environment variables. Verify that these details match your setup and cross-check with the events to ensure everything is functioning as expected. For example, the Pod below has the nginx container that is running and the events reflect the same. ```text Containers: nginx: Container ID: containerd://0dd33fd358ba3b369de315be15b197e369342aba7aa8d3ea12e4455823fa90ce Image: nginx-gateway-fabric/nginx:latest Image ID: sha256:e5cb19bab49cbde6222df607a0946e1e00c1af767263b79ae36e4c69f8547f20 Ports: 80/TCP, 9113/TCP Host Ports: 0/TCP, 0/TCP SeccompProfile: RuntimeDefault State: Running Started: Thu, 24 Apr 2025 10:57:36 -0700 Ready: True Restart Count: 0 Environment: Mounts: /etc/nginx-agent from nginx-agent (rw) /etc/nginx/conf.d from nginx-conf (rw) /etc/nginx/includes from nginx-includes (rw) /etc/nginx/main-includes from nginx-main-includes (rw) /etc/nginx/secrets from nginx-secrets (rw) /etc/nginx/stream-conf.d from nginx-stream-conf (rw) /var/cache/nginx from nginx-cache (rw) /var/lib/nginx-agent from nginx-agent-lib (rw) /var/log/nginx-agent from nginx-agent-log (rw) /var/run/nginx from nginx-run (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f9kph (ro) /var/run/secrets/ngf from nginx-agent-tls (rw) /var/run/secrets/ngf/serviceaccount from token (rw) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m57s default-scheduler Successfully assigned default/gateway-nginx-85f7f6d7d-fx7q2 to kind-control-plane Normal Pulled 2m54s kubelet Container image "nginx-gateway-fabric:latest" already present on machine Normal Created 2m54s kubelet Created container: init Normal Started 2m54s kubelet Started container init Normal Pulled 2m53s kubelet Container image "nginx-gateway-fabric/nginx:latest" already present on machine Normal Created 2m53s kubelet Created container: nginx Normal Started 2m53s kubelet Started container nginx ``` ##### NGINX Plus failure to start or traffic interruptions Beginning with NGINX Gateway Fabric 1.5.0, NGINX Plus requires a valid JSON Web Token (JWT) to run. If this is not set up properly, or your JWT token has expired, you may see errors in the NGINX logs that look like the following: ```text nginx: [error] invalid license token ``` ```text nginx: [emerg] License file is required. Download JWT license from MyF5 and configure its location... ``` ```text nginx: [emerg] license expired ``` These errors could prevent NGINX Plus from starting or prevent traffic from flowing. To fix these issues, see the [NGINX Plus JWT](/ngf/install/nginx-plus.md) guide. ##### 413 Request Entity Too Large If you receive the following error: ```text 413 Request Entity Too Large

413 Request Entity Too Large

nginx/1.25.5
``` Or view the following error message in the NGINX logs: ```text 2024/05/30 21:48:22 [error] 138#138: *43 client intended to send too large body: 112 bytes, client: 127.0.0.1, server: cafe.example.com, request: "POST /coffee HTTP/1.1", host: "cafe.example.com:8080" ``` The request body exceeds the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). To **resolve** this, you can configure the `client_max_body_size` using the `ClientSettingsPolicy` API. Read the [Client Settings Policy](/ngf/traffic-management/client-settings.md) documentation for more information. ##### IP Family Mismatch Errors If you `describe` your HTTPRoute and see the following error: ```text Conditions: Last Transition Time: 2024-07-14T23:36:37Z Message: The route is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2024-07-14T23:36:37Z Message: Service configured with IPv4 family but NginxProxy is configured with IPv6 Observed Generation: 1 Reason: InvalidServiceIPFamily Status: False Type: ResolvedRefs Controller Name: gateway.nginx.org/nginx-gateway-controller ``` The Service associated with your HTTPRoute is configured with a IP Family different than the one specified in the NginxProxy configuration. To **resolve** this, you can do one of the following: - Update the NginxProxy configuration with the proper [`ipFamily`](/ngf/reference/api.md) field. You can edit the NginxProxy configuration using `kubectl edit`. For example: ```shell kubectl edit -n nginx-gateway nginxproxies.gateway.nginx.org ngf-proxy-config ``` - When installing NGINX Gateway Fabric, change the IPFamily by modifying the field `nginx.config.ipFamily` in the `values.yaml` or add the `--set nginx.config.ipFamily=` flag to the `helm install` command. The supported IPFamilies are `ipv4`, `ipv6` and `dual` (default). - Adjust the IPFamily of your Service to match that of the NginxProxy configuration. ##### Policy cannot be applied to target If you `describe` your Policy and see the following error: ```text Conditions: Last Transition Time: 2024-08-20T14:48:53Z Message: Policy cannot be applied to target "default/route1" since another Route "default/route2" shares a hostname:port/path combination with this target Observed Generation: 3 Reason: TargetConflict Status: False Type: Accepted ``` This means you are attempting to attach a Policy to a Route that has an overlapping hostname:port/path combination with another Route. To work around this, you can do one of the following: - Combine the Route rules for the overlapping path into a single Route. - If the Policy allows it, specify both Routes in the `targetRefs` list. ##### Broken Header error If you check your _nginx_ container logs and see the following error: ```text 2024/07/25 00:50:45 [error] 211#211: *22 broken header: "GET /coffee HTTP/1.1" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:80 ``` It indicates that `proxy_protocol` is enabled for the gateway listeners, but the request sent to the application endpoint does not contain proxy information. To **resolve** this, you can do one of the following: - Unassign the field [`rewriteClientIP.mode`](/ngf/reference/api.md) in the NginxProxy configuration. - Send valid proxy information with requests being handled by your application. #### Custom Resource Definition is invalid due to being too large If you apply the NGINX Gateway Fabric CRDs and see the following error: ```text The CustomResourceDefinition "nginxproxies.gateway.nginx.org" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes ``` This is a known issue with Kubernetes, more specifically the client-side-apply annotation limit with kubectl, and our recommended workaround is to use kubectl's server-side apply when installing the NGINX Gateway Fabric CRDs. To do this, you can use the `--server-side` flag when applying the CRDs: ```shell kubectl apply --server-side -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/deploy/crds.yaml ``` ### See also You can view the [Kubernetes Troubleshooting Guide](https://kubernetes.io/docs/tasks/debug/debug-application/) for more debugging guidance. -------------------------------------------------------------------------------- title: "Upgrade applications without downtime" url: https://docs.nginx.com/nginx-gateway-fabric/how-to/upgrade-apps-without-downtime/index.md -------------------------------------------------------------------------------- Learn how to use NGINX Gateway Fabric to upgrade applications without downtime. --- ## Overview **Note:** See the [Architecture document](/ngf/overview/gateway-architecture.md) to learn more about NGINX Gateway Fabric architecture. NGINX Gateway Fabric allows upgrading applications without downtime. To understand the upgrade methods, you need to be familiar with the NGINX features that help prevent application downtime: Graceful configuration reloads and upstream server updates. --- ### Graceful configuration reloads If a relevant gateway API or built-in Kubernetes resource is changed, NGINX Gateway Fabric will update NGINX by regenerating the NGINX configuration. NGINX Gateway Fabric then sends a reload signal to the master NGINX process to apply the new configuration. We call such an operation a "reload", during which client requests are not dropped - which defines it as a graceful reload. This process is further explained in the [NGINX configuration documentation](https://nginx.org/en/docs/control.html?#reconfiguration). --- ### Upstream server updates Endpoints frequently change during application upgrades: Kubernetes creates pods for the new version of an application and removes the old ones, creating and removing the respective endpoints as well. NGINX Gateway Fabric detects changes to endpoints by watching their corresponding [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/). In an NGINX configuration, a service is represented as an [upstream](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream), and an endpoint as an [upstream server](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#server). Adding and removing endpoints are two of the most common cases: - If an endpoint is added, NGINX Gateway Fabric adds an upstream server to NGINX that corresponds to the endpoint, then reloads NGINX. Next, NGINX will start proxying traffic to that endpoint. - If an endpoint is removed, NGINX Gateway Fabric removes the corresponding upstream server from NGINX. After a reload, NGINX will stop proxying traffic to that server. However, it will finish proxying any pending requests to that server before switching to another endpoint. As long as you have more than one endpoint ready, clients won't experience downtime during upgrades. **Note:** It is good practice to configure a [Readiness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) in the deployment so that a pod can report when it is ready to receive traffic. Note that NGINX Gateway Fabric will not add any endpoint to NGINX that is not ready. --- ## Prerequisites - You have deployed your application as a [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) - The pods of the deployment belong to a [service](https://kubernetes.io/docs/concepts/services-networking/service/) so that Kubernetes creates an [endpoint](https://kubernetes.io/docs/reference/kubernetes-api/service-resources/endpoints-v1/) for each pod. - You have exposed the application to the clients via an [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) resource that references that service. For example, an application can be exposed using a routing rule like below: ```yaml - matches: - path: type: PathPrefix value: / backendRefs: - name: my-app port: 80 ``` **Note:** See the [Cafe example](https://github.com/nginx/nginx-gateway-fabric/tree/v/examples/cafe-example) for a basic example. The upgrade methods in the next sections cover: - Rolling deployment upgrades - Blue-green deployments - Canary releases --- ## Rolling deployment upgrade To start a [rolling deployment upgrade](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment), you update the deployment to use the new version tag of the application. As a result, Kubernetes terminates the pods with the old version and create new ones. By default, Kubernetes also ensures that some number of pods always stay available during the upgrade. This upgrade will add new upstream servers to NGINX and remove the old ones. As long as the number of pods (ready endpoints) during an upgrade does not reach zero, NGINX will be able to proxy traffic, and therefore prevent any downtime. This method does not require you to update the **HTTPRoute**. --- ## Blue-green deployments With this method, you deploy a new version of the application (blue version) as a separate deployment, while the old version (green) keeps running and handling client traffic. Next, you switch the traffic from the green version to the blue. If the blue works as expected, you terminate the green. Otherwise, you switch the traffic back to the green. There are two ways to switch the traffic: - Update the service selector to select the pods of the blue version instead of the green. As a result, NGINX Gateway Fabric removes the green upstream servers from NGINX and adds the blue ones. With this approach, it is not necessary to update the **HTTPRoute**. - Create a separate service for the blue version and update the backend reference in the **HTTPRoute** to reference this service, which leads to the same result as with the previous option. --- ## Canary releases Canary releases involve gradually introducing a new version of your application to a subset of nodes in a controlled manner, splitting the traffic between the old are new (canary) release. This allows for monitoring and testing the new release's performance and reliability before full deployment, helping to identify and address issues without impacting the entire user base. To support canary releases, you can implement an approach with two deployments behind the same service (see [Canary deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#canary-deployment) in the Kubernetes documentation). However, this approach lacks precision for defining the traffic split between the old and the canary version. You can greatly influence it by controlling the number of pods (for example, four pods of the old version and one pod of the canary). However, note that NGINX Gateway Fabric uses [`random two least_conn`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#random) load balancing method, which doesn't guarantee an exact split based on the number of pods (80/20 in the given example). A more flexible and precise way to implement canary releases is to configure a traffic split in an **HTTPRoute**. In this case, you create a separate deployment for the new version with a separate service. For example, for the rule below, NGINX will proxy 95% of the traffic to the old version endpoints and only 5% to the new ones. ```yaml - matches: - path: type: PathPrefix value: / backendRefs: - name: my-app-old port: 80 weight: 95 - name: my-app-new port: 80 weight: 5 ``` **Note:** Every request coming from the same client won't necessarily be sent to the same backend. NGINX will independently split each request among the backend references. By updating the rule you can further increase the share of traffic the new version gets and finally completely switch to the new version: ```yaml - matches: - path: type: PathPrefix value: / backendRefs: - name: my-app-old port: 80 weight: 0 - name: my-app-new port: 80 weight: 1 ``` See the [Traffic splitting example](https://github.com/nginx/nginx-gateway-fabric/tree/v/examples/traffic-splitting) from our repository. -------------------------------------------------------------------------------- title: "Configure Cross-Origin Request Sharing (CORS)" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-security/cors/index.md -------------------------------------------------------------------------------- This document describes how to configure the HTTPCORSFilter in F5 NGINX Gateway Fabric to handle Cross-Origin Resource Sharing (CORS) for your applications. CORS is a security feature that allows or denies web applications running at one domain to make requests for resources from a different domain. The HTTPCORSFilter in the gateway API provides a standard way to configure CORS policies. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Deploy sample application To deploy the `coffee` application, run the following YAML with `kubectl apply`: ```yaml kubectl apply -f - < ``` ## Deploy a HTTPRoute with the HTTPCORSFilter In this example, the filter is applied to the `/coffee` path. Run the following command to apply the route: ```yaml kubectl apply -f - < ``` ## Verify CORS pre-flight check **Note:** Your clients should be able to resolve the domain name "cafe.example.com" to the public IP of the NGINX Service. This guide simulates it using curl's `--resolve` option. Send a preflight request using curl: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -H "Origin: https://example.com" -X OPTIONS -v ``` Response: ```text > OPTIONS /coffee HTTP/1.1 > Host: cafe.example.com:8080 > User-Agent: curl/8.7.1 > Accept: */* > Origin: https://example.com > * Request completely sent off < HTTP/1.1 200 OK < Server: nginx < Date: Wed, 18 Feb 2026 11:49:23 GMT < Content-Type: application/octet-stream < Content-Length: 0 < Connection: keep-alive < Access-Control-Allow-Origin: https://example.com < Access-Control-Allow-Methods: GET, POST < Access-Control-Allow-Headers: Keep-Alive, Content-Type, User-Agent, Authorization < Access-Control-Expose-Headers: Content-Security-Policy < Access-Control-Allow-Credentials: true < Access-Control-Max-Age: 10 < * Connection #0 to host cafe.example.com left intact ``` ## Further reading - [Example deployment files for HTTPCORSFilter](https://github.com/nginx/nginx-gateway-fabric/tree/main/examples/cors-filter) - [Gateway API Specification](https://gateway-api.sigs.k8s.io/reference/spec/#httpcorsfilter) - [Gateway API CORS](https://gateway-api.sigs.k8s.io/guides/http-cors/) -------------------------------------------------------------------------------- title: "Configure Request Mirroring" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/mirror/index.md -------------------------------------------------------------------------------- Learn how to mirror your HTTP or gRPC traffic using NGINX Gateway Fabric. ## Overview [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/) and [GRPCRoute](https://gateway-api.sigs.k8s.io/api-types/grpcroute/) filters can be used to configure request mirroring. Mirroring copies a request to another backend. In this guide, we will set up two applications, **coffee** and **tea**, and mirror requests between them. All requests sent to the **coffee** application will also be sent to the **tea** application automatically. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Set up Create the **coffee** and **tea** applications in Kubernetes by copying and pasting the following block into your terminal: ```yaml kubectl apply -f - < 80/TCP 3s service/tea ClusterIP 10.96.185.235 80/TCP 3s ``` --- ## Configure request mirroring First, create the **cafe** Gateway resource: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Now create an HTTPRoute that defines a RequestMirror filter that copies all requests sent to `/coffee` to be sent to the **coffee** backend and mirrored to the **tea** backend. Use the following command: ```yaml kubectl apply -f - <` when running `helm install` or `helm upgrade`. To upgrade the release with Helm, you can use the OCI registry, or download the chart and upgrade from the source. If needed, replace `ngf` with your chosen release name. #### Upgrade from the OCI registry To avoid downtime when upgrading from v2.0.x to v2.1, run the following commands. Be sure to include your previous installation flags and values if necessary. This will not affect user traffic, as the NGINX data plane deployment won't be removed as part of this process. ```shell helm uninstall ngf -n nginx-gateway helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric -n nginx-gateway ``` Otherwise, for all other version upgrades: ```shell helm upgrade ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric -n nginx-gateway ``` #### Upgrade from sources ```shell helm pull oci://ghcr.io/nginx/charts/nginx-gateway-fabric --untar cd nginx-gateway-fabric ``` For the latest version from the **main** branch, add _--version 0.0.0-edge_ to your pull command. To avoid downtime when upgrading from v2.0.x to v2.1, run the following. Be sure to include your previous installation flags and values if necessary. This will not affect user traffic, as the NGINX data plane deployment won't be removed as part of this process. ```shell helm uninstall ngf -n nginx-gateway helm install ngf . -n nginx-gateway ``` Otherwise, for all other version upgrades: ```shell helm upgrade ngf . -n nginx-gateway ``` #### Manifests Select the deployment manifest that matches your current deployment from options available in the [Deploy NGINX Gateway Fabric](/ngf/install/manifests.md#deploy-nginx-gateway-fabric-1) section. To avoid downtime when upgrading from v2.0.x to v2.1, delete the previous NGINX Gateway Fabric control plane deployment in the `nginx-gateway` namespace, using `kubectl delete deployment`. Then `kubectl apply` the updated manifest file. This will not affect user traffic, as the NGINX data plane deployment won't be removed as part of this process. ## Upgrade from v1.x to v2.x This section provides step-by-step instructions for upgrading NGINX Gateway Fabric from version 1.x to 2.x, highlighting key architectural changes, expected downtime, and important considerations for CRDs. To upgrade NGINX Gateway Fabric from version 1.x to the new architecture in version 2.x, you must uninstall the existing NGINX Gateway Fabric CRDs and deployment, and perform a fresh installation. This will cause brief downtime during the upgrade process. **Note:** You do not need to uninstall the Gateway API CRDs during the upgrade. These resources are compatible with the new NGINX Gateway Fabric version. ### Uninstall NGINX Gateway Fabric v1.x To remove the previous version 1.x of NGINX Gateway Fabric, follow these steps: First, run the following command to uninstall NGINX Gateway Fabric from the `nginx-gateway` namespace, and update `ngf` to your release name if it is different: ```shell helm uninstall ngf -n nginx-gateway ``` Afterwards, remove CRDs associated with NGINX Gateway Fabric version 1.x with the following command: ```shell kubectl delete -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.2/deploy/crds.yaml ``` ### Install NGINX Gateway Fabric 2.x **Note:** Before installing 2.x, we recommend following [Add certificates for secure authentication](/ngf/install/secure-certificates.md). By default, NGINX Gateway Fabric installs self-signed certificates, which may be unsuitable for a production environment. #### Helm Use the following `helm install` command to install the latest stable NGINX Gateway Fabric release in the `nginx-gateway` namespace. It will also install the CRDs required for the deployment: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway ``` For customization options during the Helm installation process, view the [Install NGINX Gateway Fabric with Helm](/ngf/install/helm.md) topic. #### Manifests Apply the new CRDs with the following command: ```shell kubectl apply --server-side -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/deploy/crds.yaml ``` Next, install the latest stable release of NGINX Gateway Fabric in the `nginx-gateway` namespace with the following command: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/deploy/default/deploy.yaml ``` For customization options during the Manifest installation process, view the [Install NGINX Gateway Fabric with Manifests](/ngf/install/manifests.md) topic. ### Architecture changes With this release, NGINX Gateway Fabric adopts a new architecture that separates the control plane and data plane into independent deployments. This separation improves scalability, security, and operational clarity. The control plane is a Kubernetes controller that watches Gateway API and Kubernetes resources (e.g., Services, Endpoints, Secrets) and dynamically provisions NGINX data plane deployments for each Gateway. NGINX configurations are generated by the control plane and securely delivered to the data planes via gRPC, using the NGINX Agent. TLS is enabled by default, with optional integration with `cert-manager`. Each data plane pod runs NGINX alongside the Agent, which applies config updates and handles reloads without shared volumes or signals. This design ensures dynamic, per-Gateway traffic management and operational isolation. New fields have been added to the `NginxProxy` resource to configure infrastructure-related settings for data plane deployments. The `NginxProxy` resource is now a namespaced-scoped resource, instead of a cluster-scoped resource, and can be modified at either the Gateway or GatewayClass level. These new fields provide the flexibility to customize deployment and service configurations. For detailed instructions on how to modify these settings, refer to the [Configure infrastructure-related settings](/ngf/how-to/data-plane-configuration.md#configure-infrastructure-related-settings) guide. ### Key links for the version 2.x update - To read more on [modifying data plane configuration](/ngf/how-to/data-plane-configuration.md). - To learn more about [deploying a Gateway for data plane instances](/ngf/install/deploy-data-plane.md). - To add secure [authentication to control plane and data planes](/ngf/install/secure-certificates.md). - To read more about [architecture changes](/ngf/overview/gateway-architecture.md). - For detailed [API reference](/ngf/reference/api.md). ## Access NGINX Gateway Fabric 1.x documentation The documentation website is intended for the latest version of NGINX Gateway Fabric. To review documentation prior to 2.x, check out the desired release branch (such as _release-1.6_): ```shell git clone git@github.com:nginx/nginx-gateway-fabric.git git checkout release-1.6 ``` To review the documentation in a local webserver, run _make watch_ in the _/site_ folder: ```shell cd site make watch ``` ```text Hugo is available and has a version greater than 133. Proceeding with build. hugo --bind 0.0.0.0 -p 1313 server --disableFastRender Watching for changes in /home//nginx-gateway-fabric/site/{content,layouts,static} Watching for config changes in /home//nginx-gateway-fabric/site/config/_default, /home//nginx-gateway-fabric/site/config/development, /home//nginx-gateway-fabric/site/go.mod Start building sites … hugo v0.135.0-f30603c47f5205e30ef83c70419f57d7eb7175ab linux/amd64 BuildDate=2024-09-27T13:17:08Z VendorInfo=gohugoio | EN -------------------+------ Pages | 72 Paginator pages | 0 Non-page files | 0 Static files | 176 Processed images | 0 Aliases | 9 Cleaned | 0 Built in 213 ms Environment: "development" Serving pages from disk Web Server is available ``` You can then follow [this localhost link](http://localhost:1313/nginx-gateway-fabric/) for 1.x NGINX Gateway Fabric documentation. ## Upgrade from NGINX Open Source to NGINX Plus **Note:** Ensure that you [Set up the JWT](/ngf/install/nginx-plus.md#set-up-the-jwt) before upgrading. These instructions only apply to Helm. To upgrade from NGINX Open Source to NGINX Plus, update the Helm command to include the necessary values for Plus: **Note:** If applicable: - Replace the F5 Container registry `private-registry.nginx.com` with your internal registry for your NGINX Plus image - Replace `nginx-plus-registry-secret` with your Secret name containing the registry credentials - Replace `ngf` with your chosen release name. ```shell helm upgrade ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --set nginx.image.repository=private-registry.nginx.com/nginx-gateway-fabric/nginx-plus --set nginx.plus=true --set nginx.imagePullSecret=nginx-plus-registry-secret -n nginx-gateway ``` -------------------------------------------------------------------------------- title: "Gateway API Inference Extension" url: https://docs.nginx.com/nginx-gateway-fabric/how-to/gateway-api-inference-extension/index.md -------------------------------------------------------------------------------- Learn how to use NGINX Gateway Fabric with the Gateway API Inference Extension to optimize traffic routing to self-hosting Generative AI Models on Kubernetes. ## Overview The [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) is an official Kubernetes project that aims to provide optimized load-balancing for self-hosted Generative AI Models on Kubernetes. The project's goal is to improve and standardize routing to inference workloads across the ecosystem. Coupled with the provided Endpoint Picker Service, NGINX Gateway Fabric becomes an [Inference Gateway](https://gateway-api-inference-extension.sigs.k8s.io/#concepts-and-definitions), with additional AI specific traffic management features such as model-aware routing, serving priority for models, model rollouts, and more. ## Set up Install the Gateway API Inference Extension CRDs: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/inference-extension/?ref=v" | kubectl apply -f - ``` To enable the Gateway API Inference Extension, [install](/ngf/install/) NGINX Gateway Fabric with these modifications: - Using Helm: set the `nginxGateway.gwAPIInferenceExtension.enable=true` Helm value. - Using Kubernetes manifests: set the `--gateway-api-inference-extension` flag in the nginx-gateway container argument, update the ClusterRole RBAC to add the `inferencepools`: ```yaml - apiGroups: - inference.networking.k8s.io resources: - inferencepools verbs: - get - list - watch - apiGroups: - inference.networking.k8s.io resources: - inferencepools/status verbs: - update ``` See this [example manifest](https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/main/deploy/inference/deploy.yaml) for clarification. ## Deploy a sample model server The [vLLM simulator](https://github.com/llm-d/llm-d-inference-sim/tree/main) model server does not use GPUs and is ideal for test/development environments. To deploy the vLLM simulator, run the following command: ```shell kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v/config/manifests/vllm/sim-deployment.yaml ``` ## Deploy the InferencePool and Endpoint Picker Extension The InferencePool is a Gateway API Inference Extension resource that represents a set of Inference-focused Pods. With InferencePool, you can configure a routing extension as well as inference-specific routing optimizations. For more information on this resource, refer to the Gateway API Inference Extension [InferencePool documentation](https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/). Install an InferencePool named `vllm-qwen3-32b` that selects from endpoints with label `app: vllm-qwen3-32b` and listening on port 8000. The Helm install command automatically installs the Endpoint Picker Extension and InferencePool. NGINX will query the Endpoint Picker Extension to determine the appropriate pod endpoint to route traffic to. These pods are selected from a pool of ready pods designated by the assigned InferencePool's Selector field. For more information on the [Endpoint Picker](https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/README.md). **Note:** The Endpoint Picker Extension is a third-party application written and provided by the Gateway API Inference Extension project. Communication between NGINX and the Endpoint Picker uses TLS with certificate verification disabled by default. NGINX Gateway Fabric is not responsible for any threats or risks associated with using this third-party Endpoint Picker Extension application. ```shell export IGW_CHART_VERSION=v helm install vllm-qwen3-32b \ --dependency-update \ --set inferencePool.modelServers.matchLabels.app=vllm-qwen3-32b \ --version $IGW_CHART_VERSION \ --set inferenceExtension.resources.requests.memory=4Gi \ oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool ``` **Note:** For test environments, lower the CPU and memory requests and limits to reduce resource use: ```shell export IGW_CHART_VERSION=v helm install vllm-qwen3-32b \ --dependency-update \ --set inferencePool.modelServers.matchLabels.app=vllm-qwen3-32b \ --version $IGW_CHART_VERSION \ --set inferenceExtension.resources.requests.cpu=100m \ --set inferenceExtension.resources.requests.memory=512Mi \ --set inferenceExtension.resources.limits.memory=2Gi \ oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool ``` Confirm that the Endpoint Picker was deployed and is running: ```shell kubectl describe deployment vllm-qwen3-32b-epp ``` ## Deploy an Inference Gateway ```yaml kubectl apply -f - < ``` ## Deploy an HTTPRoute ```yaml kubectl apply -f - < > gateway-api-resources.yaml ``` This tool will analyze your Ingress resources from the input file and output the equivalent Gateway API resources to a file named `gateway-api-resources.yaml`. Review the generated Gateway API resources in the output file and apply them to your cluster: ```shell kubectl apply -f gateway-api-resources.yaml ``` For detailed information about NGINX-specific features and conversion options, see the [NGINX provider documentation](https://github.com/kubernetes-sigs/ingress2gateway/blob/main/pkg/i2gw/providers/nginx/README.md). -------------------------------------------------------------------------------- title: "Snippets" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/snippets/index.md -------------------------------------------------------------------------------- **Note:** This guide previously showed how to use `SnippetsFilters` to configure Rate Limiting in NGINX. However, first-class support for Rate Limiting is now supported through the `RateLimitPolicy` API. This guide has been changed to provide an example of how to use `limit_conn` to limit the number of connections to a location and `limit_except` to limit the allowed HTTP methods. For a guide on how to configure Rate Limiting, see our guide on the [RateLimitPolicy API](). This topic introduces Snippets, how to implement them using the `SnippetsFilter` and `SnippetsPolicy` APIs, and provides an example of how to use them to limit the number of connections to a location using the [limit_conn](https://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn) NGINX directive and limit the allowed HTTP methods inside a location using the [limit_except](https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_except) NGINX directive. ## Overview Snippets allow users to insert NGINX configuration into different contexts of the NGINX configurations that NGINX Gateway Fabric generates. Snippets should only be used by advanced NGINX users who need more control over the generated NGINX configuration, and only in cases where Gateway API resources or NGINX extension policies don't apply. Users can configure Snippets through either the `SnippetsFilter` or `SnippetsPolicy` APIs. `SnippetsFilter` can be an [HTTPRouteFilter](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.HTTPRouteFilter) or [GRPCRouteFilter](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.GRPCRouteFilter), that can be defined in an HTTPRoute/GRPCRoute rule and is intended to modify NGINX configuration specifically for that Route rule. `SnippetsFilter` is an `extensionRef` type filter, and must be in the same namespace as the Route it is referenced on. `SnippetsPolicy` is a [Direct PolicyAttachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that can only be applied to a Gateway(s) in the same namespace as the `SnippetsPolicy`. `SnippetsPolicy` affects the Gateway and all Routes attached to it, and is meant to contrast the route-focused `SnippetsFilter`. ## Disadvantages of Snippets **Note:** We recommend managing NGINX configuration through Gateway API resources, [first-class policies](/ngf/overview/custom-policies.md), and other existing [NGINX extensions](/ngf/how-to/data-plane-configuration.md) before using Snippets. Snippets are configured using the `SnippetsFilter` or `SnippetsPolicy` API, but are disabled by default due to their complexity and security implications. Snippets have the following disadvantages: - _Complexity_. Snippets require you to: - Understand NGINX configuration primitives to implement correct NGINX configuration. - Understand how NGINX Gateway Fabric generates NGINX configuration so that a Snippet doesn’t interfere with the other features in the configuration. - _Decreased robustness_. An incorrect Snippet can invalidate NGINX configuration, causing reload failures. Until the snippet is fixed, it will prevent any new configuration updates, including updates for the other Gateway resources. - _Security implications_. Snippets give access to NGINX configuration primitives, which are not validated by NGINX Gateway Fabric. For example, a Snippet can configure NGINX to serve the TLS certificates and keys used for TLS termination for Gateway resources. ## Best practices for Snippets When using both `SnippetsPolicies` and `SnippetsFilters`, be sure to double check that the Snippets applied do not conflict and cause invalid NGINX configuration. NGINX Gateway Fabric does not validate NGINX configuration in a Snippet. If the Snippet creates invalid NGINX configuration, NGINX Gateway Fabric will rollback to the latest valid configuration with a related log message outputted. ### SnippetsFilters There are endless ways to use `SnippetsFilters` to modify NGINX configuration, and equal ways to generate invalid or undesired NGINX configuration. We have outlined a few best practices to keep in mind when using `SnippetsFilters` to keep NGINX Gateway Fabric functioning correctly: 1. Using the [Roles and Personas](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/) defined in the Gateway API, `SnippetsFilter` access should be limited to Cluster operators. Application developers should not be able to create, modify, or delete `SnippetsFilters` as they affect other applications. `SnippetsFilter` creates a natural split of responsibilities between the Cluster operator and the Application developer: the Cluster operator creates a `SnippetsFilter`; the Application developer references the `SnippetsFilter` in an HTTPRoute/GRPCRoute to enable it. 1. In a `SnippetsFilter`, only one Snippet per NGINX context is allowed, however multiple `SnippetsFilters` can be referenced in the same routing rule. As such, `SnippetsFilters` should not conflict with each other. If `SnippetsFilters` do conflict, they should not be referenced on the same routing rule. 1. `SnippetsFilters` that define Snippets targeting NGINX contexts `main`, `http`, or `http.server`, can potentially affect more than the routing rule they are referenced by. Proceed with caution and verify the behavior of the NGINX configuration before creating those `SnippetsFilters` in a production scenario. ### SnippetsPolicies `SnippetsPolicies` share many of the same best practices as `SnippetsFilters` but with a couple of adjustments: 1. `SnippetsPolicy` access should be limited to Cluster operators. Additionally, since `SnippetsPolicies` use the `targetRefs` field to specify which resources to target, this means there is no need for an Application developer to reference it on a Route like the `SnippetsFilter` does. ## Setup To enable the `SnippetsFilter` and `SnippetsPolicy` APIs, [install](/ngf/install/) NGINX Gateway Fabric with these modifications: - Using Helm: set the `nginxGateway.snippets.enable=true` Helm value. - Using Kubernetes manifests: set the `--snippets` flag in the nginx-gateway container argument, add `snippetsfilters` and `snippetspolicies` to the RBAC rules with verbs `list` and `watch`, and add `snippetsfilters/status` and `snippetspolicies/status` to the RBAC rules with verb `update`. See this [example manifest](https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/main/deploy/snippets/deploy.yaml) for clarification. Create the coffee and tea example applications: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/snippets/app.yaml ``` Create a Gateway: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/snippets/gateway.yaml ``` After creating the Gateway resource, NGINX Gateway Fabric will provision an NGINX Pod and Service fronting it to route traffic. Verify the gateway is created: ```shell kubectl describe gateways.gateway.networking.k8s.io gateway ``` Verify the status is `Accepted`: ```text Status: Addresses: Type: IPAddress Value: 10.96.36.219 Conditions: Last Transition Time: 2026-01-09T05:40:37Z Message: The Gateway is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2026-01-09T05:40:37Z Message: The Gateway is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed ``` Save the public IP address and port(s) of the Gateway into shell variables: ```text GW_IP=XXX.YYY.ZZZ.III GW_PORT= ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Create HTTPRoutes for the coffee and tea applications: ```shell kubectl apply -f - < ``` ### Configure coffee HTTPRoute to reference SnippetsFilter To use the `limit-except-sf` `SnippetsFilter`, update the coffee HTTPRoute to reference it: ```yaml kubectl apply -f - < 403 Forbidden

403 Forbidden

nginx
``` Since the `SnippetsFilter` was only referenced on the coffee HTTPRoute and the context was `http.server.location`, the tea HTTPRoute should not be affected. To verify, send a `POST` request to tea: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea -POST ``` ```text Server address: 10.244.0.36:8080 Server name: tea-75bc9f4b6d-9msq7 Date: 21/Jan/2026:21:09:52 +0000 URI: /tea Request ID: 1fa64c2795e476534d2d8d6759a85c4c ``` ## Create a SnippetsPolicy Create a `SnippetsPolicy` named `limit-conn-sp` by adding the following `SnippetsPolicy`: ```yaml kubectl apply -f - < ``` Verify that the Gateway has been configured correctly: ```shell kubectl describe gateway gateway ``` You should see the following condition: ```text Status: Conditions: Last Transition Time: 2026-01-17T00:50:58Z Message: The SnippetsPolicy is applied to the resource Observed Generation: 1 Reason: PolicyAffected Status: True Type: SnippetsPolicyAffected ``` ### Send multiple requests in parallel The `SnippetsPolicy` targets the Gateway, meaning all attached Routes should be affected. This means that both the coffee and tea applications should be affected and have their maximum concurrent connections set to one. To verify the `SnippetsPolicy` is working, we will send multiple curl requests in parallel. Send requests to the coffee application: ```shell curl --parallel --parallel-immediate \ --resolve cafe.example.com:8080:127.0.0.1 "http://cafe.example.com:8080/coffee" \ --resolve cafe.example.com:8080:127.0.0.1 "http://cafe.example.com:8080/coffee" \ --resolve cafe.example.com:8080:127.0.0.1 "http://cafe.example.com:8080/coffee" ``` You should see that only a single request had a successful response while the others had `503 Service Temporarily Unavailable` status codes returned: ```text ... 503 Service Temporarily Unavailable

503 Service Temporarily Unavailable

nginx
Server address: 10.244.0.35:8080 Server name: coffee-654ddf664b-kmzzs Date: 21/Jan/2026:21:19:55 +0000 URI: /coffee Request ID: 1ffcc0fae174ea95da472c6984466ae5 ``` Next, verify the same behavior in the tea application: Send requests to the tea application: ```shell curl --parallel --parallel-immediate \ --resolve cafe.example.com:8080:127.0.0.1 "http://cafe.example.com:8080/tea" \ --resolve cafe.example.com:8080:127.0.0.1 "http://cafe.example.com:8080/tea" \ --resolve cafe.example.com:8080:127.0.0.1 "http://cafe.example.com:8080/tea" ``` The results should be the same: ```text ... 503 Service Temporarily Unavailable

503 Service Temporarily Unavailable

nginx
Server address: 10.244.0.36:8080 Server name: tea-75bc9f4b6d-9msq7 Date: 21/Jan/2026:21:23:07 +0000 URI: /tea Request ID: dfb42090e331b6ae31ed894b9b35d38d ``` This shows the `SnippetsPolicy` affecting all of the Gateway, in contrast to the `SnippetsFilter` that only affects the coffee HTTPRoute. ## Conclusion Snippets are a powerful tool to modifying NGINX configuration unavailable in first-class policies. This example showed a few small simple Snippets, but Snippets can contain any valid NGINX configuration, allowing users to fully customize configuration generated by NGINX Gateway Fabric and shape their NGINX configuration to fit their needs. However, as Snippets grow in complexity, because NGINX Gateway Fabric does not provide validation, the risk of generating invalid configuration due to conflicts in existing configuration grows and debugging can be challenging. This is one reason to try to keep Snippet configuration minimal, and to transition to using newly supported first-class polices and APIs if they provide the same NGINX configuration. ## Troubleshooting If a `SnippetsFilter` or `SnippetsPolicy` is applied with a Snippet which includes an invalid NGINX configuration, NGINX will continue to operate with the last valid configuration and an event with the error will be outputted. No new configuration will be applied until the invalid Snippet is fixed. An example of an error from the NGINX Gateway Fabric `nginx-gateway` container logs: ```text {"level":"error","ts":"2026-01-21T21:30:15Z","logger":"eventHandler","msg":"Failed to update NGINX configuration","error":"msg: Config apply failed, rolling back config; error: failed to parse config invalid number of arguments in \"limit_conn\" directive in /etc/nginx/includes/SnippetsPolicy_location_default-limit-conn-sp.conf:3","stacktrace":"github.com/nginx/nginx-gateway-fabric/v2/internal/controller.(*eventHandlerImpl).waitForStatusUpdates\n\tgithub.com/nginx/nginx-gateway-fabric/v2/internal/controller/handler.go:298"} ``` An example of an error from the NGINX Pod's `nginx` container logs: ```text time=2026-01-21T21:30:15.263Z level=ERROR msg="errors found during config apply, sending error status, rolling back config" err="failed to parse config invalid number of arguments in \"limit_conn\" directive in /etc/nginx/includes/SnippetsPolicy_location_default-limit-conn-sp.conf:3" correlation_id=8334bef4-44b3-4d13-93a9-a5fe84e7f351 server_type=command ``` If a Gateway is affected by a `SnippetsFilter` or `SnippetsPolicy` that creates invalid NGINX configuration, it may also contain information in its conditions describing the error: ```text Conditions: Last Transition Time: 2026-01-18T21:58:47Z Message: The Gateway is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2026-01-18T21:58:47Z Message: The Gateway is not programmed due to a failure to reload nginx with the configuration: msg: Config apply failed, rolling back config; error: failed to parse config invalid number of arguments in "limit_conn" directive in /etc/nginx/includes/SnippetsPolicy_location_default-limit-conn-sp.conf:3 Observed Generation: 1 Reason: Invalid Status: False Type: Programmed ... Listeners: Attached Routes: 2 Conditions: ... Last Transition Time: 2026-01-18T21:58:47Z Message: The Listener is not programmed due to a failure to reload nginx with the configuration: msg: Config apply failed, rolling back config; error: failed to parse config invalid number of arguments in "limit_conn" directive in /etc/nginx/includes/SnippetsPolicy_location_default-limit-conn-sp.conf:3 Observed Generation: 1 Reason: Invalid Status: False Type: Programmed ``` If a Route references a `SnippetsFilter` which cannot be resolved, the route will return a 500 HTTP error response on all requests. The Route conditions will contain information describing the error: ```text Conditions: Last Transition Time: 2026-01-18T22:01:13Z Message: The Route is accepted Observed Generation: 3 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2026-01-18T22:01:13Z Message: Spec.rules[0].filters[0].extensionRef: Not found: {"group":"gateway.nginx.org","kind":"SnippetsFilter","name":"limit-except-sf"} Observed Generation: 3 Reason: InvalidFilter Status: False Type: ResolvedRefs ``` **Note:** If you run into situations where an NGINX directive fails to be applied and the troubleshooting information here isn't sufficient, please create an issue in the [NGINX Gateway Fabric Github repository](https://github.com/nginx/nginx-gateway-fabric). ## See also - [API reference](/ngf/reference/api.md): all configuration fields for the `SnippetsFilter` and `SnippetsPolicy` APIs. -------------------------------------------------------------------------------- title: "Support" url: https://docs.nginx.com/nginx-gateway-fabric/support/index.md -------------------------------------------------------------------------------- F5 NGINX Gateway Fabric adheres to the support policy detailed in the following knowledge base article: [K000140156](https://my.f5.com/manage/s/article/K000140156). After opening a support ticket, F5 staff will request additional information to better understand the problem. ## Kubernetes support plugin The [nginx-supportpkg-for-k8s](https://github.com/nginx/nginx-supportpkg-for-k8s) plugin collects the information needed by F5 Technical Support to assist with troubleshooting your issue. The plugin uses [krew](https://krew.sigs.k8s.io), the plugin manager for the Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/) command-line tool. The plugin may collect some or all of the following global and namespace-specific information: - Kubernetes version and information about Nodes and Custom Resources - Kubernetes metrics - Helm deployments - List of Pods, Events, ConfigMaps, Services, Deployments, Daemonsets, StatefulSets, ReplicaSets, and Leases - Pod log output - `nginx -T` output from NGINX-related Pods This plugin **does not** collect secrets or coredumps. Visit the [project’s GitHub repository](https://github.com/nginx/nginx-supportpkg-for-k8s) for further details. ## Support channels - If you experience issues with NGINX Gateway Fabric, please [open an issue](https://github.com/nginx/nginx-gateway-fabric/issues/new?assignees=&labels=&projects=&template=bug_report.md&title=) in GitHub. - If you have any suggestions or enhancement requests, please [open an idea](https://github.com/nginx/nginx-gateway-fabric/discussions/categories/ideas) on GitHub discussions. - You can also get help through the [NGINX Community Forum](https://community.nginx.org/). - If you need dedicated support for NGINX Gateway Fabric, or you would like to leverage our [advanced NGINX Plus features](https://docs.nginx.com/nginx-gateway-fabric/overview/nginx-plus/), you can contact [F5 Sales](https://www.f5.com/products/get-f5). -------------------------------------------------------------------------------- title: "Client Settings Policy API" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/client-settings/index.md -------------------------------------------------------------------------------- Learn how to use the `ClientSettingsPolicy` API. ## Overview The `ClientSettingsPolicy` API allows Cluster Operators and Application Developers to configure the connection behavior between the client and NGINX. The settings in `ClientSettingsPolicy` correspond to the following NGINX directives: - [`client_max_body_size`](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) - [`client_body_timeout`](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout) - [`client_body_buffer_size`](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) - [`keepalive_requests`](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests) - [`keepalive_time`](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_time) - [`keepalive_timeout`](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) - [`keepalive_min_timeout`](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_min_timeout) `ClientSettingsPolicy` is an [Inherited PolicyAttachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that can be applied to a Gateway, HTTPRoute, or GRPCRoute in the same namespace as the `ClientSettingsPolicy`. When applied to a Gateway, the settings specified in the `ClientSettingsPolicy` affect all HTTPRoutes and GRPCRoutes attached to the Gateway. This allows Cluster Operators to set defaults for all applications using the Gateway. When applied to an HTTPRoute or GRPCRoute, the settings in the `ClientSettingsPolicy` affect only the route they are applied to. This allows Application Developers to set values for their applications based on their application's behavior or requirements. Settings applied to an HTTPRoute or GRPCRoute take precedence over settings applied to a Gateway. See the [custom policies](/ngf/overview/custom-policies.md) document for more information on policies. This guide will show you how to use the `ClientSettingsPolicy` API to configure the client max body size for your applications. For all the possible configuration options for `ClientSettingsPolicy`, see the [API reference](/ngf/reference/api.md). ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. Create the coffee and tea example applications: ```yaml kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/client-settings-policy/app.yaml ``` Create a Gateway: ```yaml kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/client-settings-policy/gateway.yaml ``` After creating the Gateway resource, NGINX Gateway Fabric will provision an NGINX Pod and Service fronting it to route traffic. Create HTTPRoutes for the coffee and tea applications: ```yaml kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/client-settings-policy/httproutes.yaml ``` Verify the gateway is created: ```shell kubectl describe gateways.gateway.networking.k8s.io gateway ``` Verify the status is `Accepted`: ```text Status: Conditions: Last Transition Time: 2026-01-09T05:40:37Z Message: The Gateway is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2026-01-09T05:40:37Z Message: The Gateway is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed ``` Save the public IP address and port(s) of the Gateway into shell variables: ```text GW_IP=XXX.YYY.ZZZ.III GW_PORT= ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Test the configuration: You can send traffic to the coffee and tea applications using the external IP address and port for the NGINX Service. Send a request to coffee: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` This request should receive a response from the coffee Pod: ```text Server address: 10.244.0.9:8080 Server name: coffee-76c7c85bbd-cf8nz ``` Send a request to tea: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` This request should receive a response from the tea Pod: ```text Server address: 10.244.0.9:8080 Server name: tea-76c7c85bbd-cf8nz ``` ## Configure client max body size ### Set a default client max body size for the Gateway To set a default client max body size for the Gateway created during setup, add the following `ClientSettingsPolicy`: ```yaml kubectl apply -f - < ``` Next, test that the policy is configured by sending a POST request to the coffee and tea applications exceeding the client's max body size of 50 bytes. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -X POST --data "this payload is greater than fifty bytes by four bytes" ``` You should receive the following error: ```text 413 Request Entity Too Large

413 Request Entity Too Large

nginx/1.25.5
``` Try again with a payload that's less than the 50 byte limit: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -X POST --data "this payload is under fifty bytes" ``` This time, you should receive a response from coffee: ```text Server address: 10.244.0.6:8080 Server name: coffee-56b44d4c55-7ldjc ``` You can repeat this test with the tea application to confirm that the policy affects both HTTPRoutes. ### Set a different client max body size for a route To set a different client max body size for a particular route, you can create another `ClientSettingsPolicy` that targets the route: ```yaml kubectl apply -f - < ``` Notice that the Ancestor Ref in the status is the tea HTTPRoute instead of the Gateway. Next, test that the policy is configured by sending a POST request to the tea application with a request body size greater than 50 bytes. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea -X POST --data "this payload is greater than fifty bytes but less than seventy five" ``` You should receive a response from tea: ```text Server address: 10.244.0.7:8080 Server name: tea-596697966f-bf6tw ``` However, since the coffee app is still affected by the `ClientSettingsPolicy` attached to the Gateway, the same request to coffee should fail: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee -X POST --data "this payload is greater than fifty bytes but less than seventy five" ``` ```text 413 Request Entity Too Large

413 Request Entity Too Large

nginx/1.25.5
``` To configure a `ClientSettingsPolicy` for a GRPCRoute, you can specify the GRPCRoute in the `spec.targetRef`: ```yaml kubectl apply -f - <" EOF ``` ## Configure the Guardrails backend Service The Guardrails backend can live outside or inside the cluster. NGINX Gateway Fabric picks the URL scheme from the referenced Service's type: | Backend location | Service type | Resolved URL | | ---------------- | ------------ | ------------ | | External | `ExternalName` | `https://:` | | In-cluster | `ClusterIP` (or any non-`ExternalName`) | `http://..svc.cluster.local:` | **note:** The `cluster.local` suffix in the in-cluster URL is the cluster's DNS domain. If your cluster uses a different domain, configure it with the `--cluster-domain` flag or `clusterDomain` Helm value when deploying NGINX Gateway Fabric (default: `cluster.local`). #### External To configure a Guardrails backend Service which is external, create an `ExternalName` Service pointing at your hosted Guardrails API: ```yaml kubectl apply -f - < ports: - name: https port: 443 protocol: TCP EOF ``` #### In-cluster For an in-cluster backend, your AI Guardrail backend pods will most likely have an existing Service which you can point the PayloadProcessor backendRef to, otherwise create a Service configured to expose your guardrail backends: ```yaml kubectl apply -f - <`. ## Deploy NGINX Gateway Fabric [Install](/ngf/install/) NGINX Gateway Fabric with the `PayloadProcessor` policy enabled: - Using Helm: set the `nginxGateway.payloadProcessor.enable=true` Helm value. - Using Kubernetes manifests: set the `--payload-processor` flag in the nginx-gateway container argument, and update the ClusterRole RBAC to add `payloadprocessors`: ```yaml - apiGroups: - gateway.nginx.org resources: - payloadprocessors verbs: - get - list - watch - apiGroups: - gateway.nginx.org resources: - payloadprocessors/status verbs: - update ``` ## Create a Gateway ```yaml kubectl apply -f - < ``` ## Create an HTTPRoute If you are using your own LLM, change the `backendRefs.name` and `backendRefs.port` to match the LLM's Service. ```yaml kubectl apply -f - <:$GW_PORT:$GW_IP http://:$GW_PORT/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"What is NGINX?"}' ``` ```text HTTP/1.1 200 OK Server: nginx Date: Tue, 11 Aug 2026 17:39:19 GMT Content-Type: application/json Content-Length: 607 Connection: keep-alive X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2 X-Inference-Port: 8000 {"id":"cmpl-f657bc1b-c50c-5a5c-9408-45bd22a150a1","created":1786469959,"model":"meta-llama/Llama-3.1-8B-Instruct","usage":{"prompt_tokens":4,"completion_tokens":62,"total_tokens":66},"object":"text_completion","kv_transfer_params":null,"choices":[{"index":0,"finish_reason":"stop","text":"NGINX (pronounced \"engine-x\") is an open-source, high-performance web server. It functions primarily as an HTTP web server, reverse proxy, load balancer, and HTTP cache.Designed to handle thousands of concurrent connections with minimal memory usage, NGINX is an essential component of modern web infrastructure."}] ``` If the request payload contains content that your Guardrails backend is configured to block, the request never reaches the LLM and returns `HTTP 403`: ```shell curl -i --resolve :$GW_PORT:$GW_IP http://:$GW_PORT/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"My SSN is 123-45-6789"}' ``` ```text HTTP/1.1 403 Forbidden Server: nginx Date: Tue, 11 Aug 2026 17:39:50 GMT Content-Length: 139 Connection: keep-alive Content-Type: application/json {"error":{"code":"content_policy_violation","message":"Request blocked by guardrails policy.","param":null,"type":"invalid_request_error"} ``` If the model's *output* contains content that your Guardrails backend blocks, the response is withheld from the client and returns `HTTP 403` with `error.type: api_error`: ```shell curl -i --resolve :$GW_PORT:$GW_IP http://:$GW_PORT/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","stream":false,"max_tokens":128,"prompt":"Give me a test SSN"}' ``` ```text HTTP/1.1 403 Forbidden Server: nginx Date: Tue, 11 Aug 2026 17:40:36 GMT Content-Type: application/json Content-Length: 128 Connection: keep-alive X-Inference-Pod: vllm-qwen3-32b-58cfff7c9-5zlm2 X-Inference-Port: 8000 {"error":{"code":"content_policy_violation","message":"Response blocked by guardrails policy.","param":null,"type":"api_error"}} ``` For more example curl requests, view the [`examples/guardrails`](https://github.com/nginx/nginx-gateway-fabric/tree/v/examples/guardrails) `README.md` in the NGINX Gateway Fabric repository. ## Troubleshooting The `PayloadProcessor` is marked `Accepted=False` when its references cannot be resolved: | Condition | Cause | Fix | | --------- | ----- | --- | | `backend Service ... not found` | `backendRef.name`/`namespace` does not match a Service. | Apply the Guardrails backend Service; check name and namespace. | | `ExternalName service has empty ... externalName` | `ExternalName` Service with a blank `externalName`. | Set `spec.externalName`. | | `auth token Secret ... not found` | `authTokenRef` set but Secret missing. | Apply the token Secret, or remove `authTokenRef`. | | NGINX error `no resolver defined to resolve `, or guardrails requests fail against an `ExternalName` backend | No `dnsResolver` configured on the NginxProxy. | Add the `dnsResolver` block and wire it via `parametersRef`. | ## Further reading - [Scan streaming in AI Security](https://docs.aisecurity.f5.com/api-docs/scan-request-streaming.html) - [Installation](/ngf/install/): install NGINX Gateway Fabric with the `PayloadProcessor` policy enabled. - [Custom policies](/ngf/overview/custom-policies.md): learn how inherited policies attach to Gateway API resources. - [`examples/guardrails`](https://github.com/nginx/nginx-gateway-fabric/tree/v/examples/guardrails): for more information on the example used in this guide. -------------------------------------------------------------------------------- title: "Upstream Settings Policy API" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/upstream-settings/index.md -------------------------------------------------------------------------------- Learn how to use the `UpstreamSettingsPolicy` API. ## Overview The `UpstreamSettingsPolicy` API allows Application Developers to configure the behavior of a connection between NGINX and the upstream applications. The settings in `UpstreamSettingsPolicy` correspond to the following NGINX directives: - [`zone`]() - [`keepalive`]() - [`keepalive_requests`]() - [`keepalive_time`]() - [`keepalive_timeout`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout) - [`random`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#random) - [`least_conn`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_conn) - [`least_time`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_time) - [`upstream`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream) - [`ip_hash`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash) - [`hash`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#hash) - [`variables`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables) `UpstreamSettingsPolicy` is a [Direct Policy Attachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that can be applied to one or more services in the same namespace as the policy. `UpstreamSettingsPolicies` can only be applied to HTTP or gRPC services, in other words, services that are referenced by an HTTPRoute or GRPCRoute. See the [custom policies](/ngf/overview/custom-policies.md) document for more information on policies. This guide will show you how to use the `UpstreamSettingsPolicy` API to configure the load balancing method, upstream zone size and keepalives for your applications. For all the possible configuration options for `UpstreamSettingsPolicy`, see the [API reference](/ngf/reference/api.md). --- ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Setup Create the `coffee` and `tea` example applications: ```yaml kubectl apply -f - < 80/TCP 23h service/tea ClusterIP 10.244.0.15 80/TCP 23h NAME READY STATUS RESTARTS AGE pod/coffee-676c9f8944-n9g6n 1/1 Running 0 23h pod/tea-6fbfdcb95d-cf84d 1/1 Running 0 23h ``` Create a Gateway: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Create HTTPRoutes for the `coffee` and `tea` applications: ```yaml kubectl apply -f - < ``` Next, verify that the policy has been applied to the `coffee` and `tea` upstreams by inspecting the NGINX configuration: ```shell kubectl exec -it deployments/gateway-nginx -- nginx -T ``` You should see the `zone` directive in the `coffee` and `tea` upstreams both specify the size `1m`: ```text upstream default_coffee_80 { random two least_conn; zone default_coffee_80 1m; server 10.244.0.14:8080; } upstream default_tea_80 { random two least_conn; zone default_tea_80 1m; server 10.244.0.15:8080; } ``` ## Enable keepalive connections By default, the `keepalive` directive is omitted, which results in the default NGINX `keepalive` value being used. You can override this value or disable `keepAlive` entirely by configuring an UpstreamSettingsPolicy. To disable keepalive, set the connections field to 0. The following example creates an `UpstreamSettingsPolicy` that configures keepalive connections for the `coffee` Service with a value of 24: ```yaml kubectl apply -f - < ``` Next, verify that the policy has been applied to the `coffee` upstreams, by inspecting the NGINX configuration: ```shell kubectl exec -it deployments/gateway-nginx -- nginx -T ``` You should see that the `coffee` upstream has the `keepalive` directive set to 24: ```text upstream default_coffee_80 { random two least_conn; zone default_coffee_80 1m; server 10.244.0.14:8080; keepalive 24; } ``` To disable the `keepalive` directive, lets create an `UpstreamSettingsPolicy` targeting the `tea` service with value 0: ```yaml kubectl apply -f - < ``` **note:** This setting applies only when the target Service has a ClusterIP. For headless Services (ClusterIP: None) and ExternalName Services, normal endpoint resolution is used instead. Additionally, this setting is also not applied to L4/stream upstreams. View the IP address of the `coffee` Service and verify it matches the IP address in the `coffee` upstream: ```shell kubectl get service coffee ``` ```text NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE coffee ClusterIP 10.96.23.26 80/TCP 16m ``` ```shell kubectl exec -it deployments/gateway-nginx -- nginx -T ``` ```text upstream default_coffee_80 { random two least_conn; zone default_coffee_80 1m; server 10.96.23.26:80; keepalive 32; } ``` --- ## Route upstream traffic to the Service ClusterIP By default, NGINX Gateway Fabric resolves each backend Service to its individual Pod IPs and uses those as the upstream servers. Setting `useClusterIP` to `true` in an `UpstreamSettingsPolicy` configures NGINX to route to the Service's ClusterIP and port instead, so the upstream contains a single server (the Service VIP). This is useful for service mesh compatibility and for controllers or operators that require traffic to traverse the Service VIP. You can also enable this globally for all Services through the `useClusterIP` field of the `NginxProxy` resource. When both are configured for the same Service, the `UpstreamSettingsPolicy` value takes precedence. See [Data plane configuration](/ngf/how-to/data-plane-configuration.md) for the global setting. **note:** Because the upstream contains only the Service VIP as a single server, you lose NGINX's load balancing across the backend Pods. Traffic is instead load balanced by the Kubernetes Service (kube-proxy), so the load balancing and keepalive settings of an `UpstreamSettingsPolicy` no longer apply to that Service. `useClusterIP` applies only when the target Service has a ClusterIP; headless (`ClusterIP: None`) and ExternalName Services fall back to the default Pod IP resolution. To route to the ClusterIP of the `coffee` service, create the following `UpstreamSettingsPolicy`: ```yaml kubectl apply -f - < ``` Find the ClusterIP of the `coffee` service: ```shell kubectl get service coffee ``` Next, verify that the `coffee` upstream targets that ClusterIP by inspecting the NGINX configuration: ```shell kubectl exec -it deployments/gateway-nginx -- nginx -T ``` You should see a single `server` in the `coffee` upstream set to the Service ClusterIP and port (`10.244.0.14` is the ClusterIP in this example): ```nginx upstream default_coffee_80 { random two least_conn; zone default_coffee_80 512k; server 10.244.0.14:80; keepalive 16; } ``` --- ## Further reading - [Custom policies](/ngf/overview/custom-policies.md): learn about how NGINX Gateway Fabric custom policies work. - [API reference](/ngf/reference/api.md): all configuration fields for the `UpstreamSettingsPolicy` API. -------------------------------------------------------------------------------- title: "Session Persistence" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/session-persistence/index.md -------------------------------------------------------------------------------- Learn how to configure session persistence using NGINX Gateway Fabric. ## Overview In this guide, you’ll learn how to configure session persistence for your application. Session persistence ensures that multiple requests from the same client are consistently routed to the same backend Pod. This is useful when your application maintains in-memory state (for example, shopping carts or user sessions). NGINX Gateway Fabric supports configuring session persistence via `UpstreamSettingsPolicy` resource or directly on `HTTPRoute` and `GRPCRoute` resources. For NGINX OSS users, using the `ip_hash` load-balancing method provides basic session affinity by routing requests from the same client IP to the same backend Pod. For NGINX Plus users, cookie-based session persistence can be configured using the `sessionPersistence` field in a Route. In this guide, you will deploy three applications: - An application configured with `ip_hash` load-balancing method. - An application configured with cookie–based session persistence (if you have access to NGINX Plus). - A regular application with default load-balancing. These applications will showcase the benefits of session persistence for stateful workloads. The NGINX directives discussed in this guide are: - [`ip_hash`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash) - [`sticky cookie`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky) ## Note **Note:** Cookie-based `SessionPersistence` is only available for [NGINX Plus](/ngf/install/nginx-plus.md) users, with alternatives provided for NGINX OSS users. Session Persistence is a Gateway API field from the experimental release channel and is subject to change. ## Before you begin [Install](/ngf/install/nginx-plus.md) NGINX Gateway Fabric with **NGINX Plus** and experimental features enabled if you want to use cookie-based `sessionPersistence`. If you plan to use the `ip_hash` load-balancing method for session affinity instead, installing NGINX Gateway Fabric with **NGINX OSS** is sufficient. To use Gateway API experimental resources, the Gateway API resources from the experimental channel must be installed before deploying NGINX Gateway Fabric. Additionally, NGINX Gateway Fabric must have experimental features enabled. **Note:** As noted in the [Gateway API documentation](https://gateway-api.sigs.k8s.io/guides/#install-experimental-channel), future releases of the Gateway API can include breaking changes to experimental resources and fields. To install the Gateway API resources from the experimental channel, run the following: ```shell kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v" | kubectl apply -f - ``` **Note:** If you plan to use the `edge` version of NGINX Gateway Fabric, you can replace the version in `ref` with `main`, for example `ref=main`. To enable experimental features on NGINX Gateway Fabric: Using Helm: Set `nginxGateway.gwAPIExperimentalFeatures.enable` to true. An example can be found in the [Installation with Helm](/ngf/install/helm.md#custom-installation-options) guide. Using Kubernetes manifests: Add the `--gateway-api-experimental-features` command-line flag to the deployment manifest args. An example can be found in the [Installation with Kubernetes manifests](/ngf/install/manifests.md#3-deploy-nginx-gateway-fabric) guide. ## Setup Create the `coffee`, `tea` and `latte` applications: ```yaml kubectl apply -f - < pod/coffee-5b9c74f9d9-7gfwn 1/1 Running 0 3h19m 10.244.0.94 kind-control-plane pod/latte-d5f64f67f-9t2j5 1/1 Running 0 3h19m 10.244.0.96 kind-control-plane pod/latte-d5f64f67f-drwc6 1/1 Running 0 3h19m 10.244.0.98 kind-control-plane pod/tea-859766c68c-cnb8n 1/1 Running 0 3h19m 10.244.0.93 kind-control-plane pod/tea-859766c68c-kttkb 1/1 Running 0 3h19m 10.244.0.97 kind-control-plane NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/coffee ClusterIP 10.96.169.1 80/TCP 3h19m app=coffee service/latte ClusterIP 10.96.42.39 80/TCP 3h19m app=latte service/tea ClusterIP 10.96.81.103 80/TCP 3h19m app=tea ``` Create a Gateway: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. ## Session Persistence Methods ## Choosing the right session persistence method for your environment The choice between `ip_hash` and cookie-based session persistence depends on your use case. The `ip_hash` load-balancing method provides basic IP-based affinity: as long as NGINX sees the real client IP and not many users share that IP, requests from the same client will usually go to the same upstream Pod. However, there are important limitations: - **Shared IPs:** When there is a load balancer or proxy in front of NGINX that does not preserve the real client IP, or when many users appear to come from a single IP address (for example, corporate NAT or VPNs), `ip_hash` operates on the shared IP rather than on individual users. This means many different users behind the same IP are all routed to the same upstream Pod, so you do not get true per-user stickiness. - **Changing IPs:** If a user’s apparent IP changes over time (for example, when a load balancer or NAT pool uses multiple egress addresses), that user can be rehashed to a different upstream Pod and lose stickiness. Cookie-based session persistence with `sticky cookie` provides stronger, per-user stickiness. NGINX issues a session cookie, and all subsequent requests that present that cookie are routed to the same upstream Pod, regardless of changes in client IP or intermediate proxies. This is generally preferable for stateful, user-centric applications, while `ip_hash` can be a simpler option in NGINX OSS deployments where NGINX sees distinct, stable client IP addresses. ### Session Persistence with NGINX OSS In this section, you’ll configure a basic `coffee` HTTPRoute that routes traffic to the `coffee` Service. You’ll then attach an `UpstreamSettingsPolicy` to change the load-balancing method for that upstream to showcase session affinity behavior. NGINX hashes the client IP to select an upstream server, so requests from the same IP are routed to the same upstream as long as it is available. Session affinity quality with `ip_hash` depends on NGINX seeing the real client IP. In environments with external load balancers or proxies, operators must ensure appropriate `real_ip_header/set_real_ip_from` configuration so that `$remote_addr` reflects the end-user address otherwise, stickiness will be determined by the address of the front-end proxy rather than the actual client. To create an HTTPRoute for the `coffee` service, copy and paste the following into your terminal: ```yaml kubectl apply -f - < 81/TCP 3m1s service/tea ClusterIP 10.96.16.136 82/TCP 3m1s ``` Create a Gateway with two TCP listeners: ```yaml kubectl apply -f - < GW_PORT_2= ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Create TCPRoutes for routing to `coffee` and `tea` applications: ```yaml kubectl apply -f - < 53/UDP 13m ``` Create a Gateway with UDP listener: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Create UDPRoute for routing to `coredns` application: ```yaml kubectl apply -f - <> DiG 9.10.6 <<>> ${GW_IP} 53 cafe.example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13260 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;localhost. IN A ;; ANSWER SECTION: localhost. 600 IN A 127.0.0.1 ;; Query time: 136 msec ;; SERVER: 192.168.72.180#53(192.168.72.180) ;; WHEN: Thu Jan 08 21:44:46 MST 2026 ;; MSG SIZE rcvd: 54 ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 32036 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;53. IN A ;; Query time: 141 msec ;; SERVER: 192.168.72.180#53(192.168.72.180) ;; WHEN: Thu Jan 08 21:44:47 MST 2026 ;; MSG SIZE rcvd: 31 ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16210 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;cafe.example.com. IN A ;; AUTHORITY SECTION: example.com. 900 IN SOA elliott.ns.cloudflare.com. dns.cloudflare.com. 2393120882 10000 2400 604800 1800 ;; Query time: 159 msec ;; SERVER: 192.168.72.180#53(192.168.72.180) ;; WHEN: Thu Jan 08 21:44:47 MST 2026 ;; MSG SIZE rcvd: 107 ``` ## Further Readings - [UDPRoute](https://gateway-api.sigs.k8s.io/reference/spec/#udproute) -------------------------------------------------------------------------------- title: "Proxy Settings Policy API" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/proxy-settings/index.md -------------------------------------------------------------------------------- Learn how to use the `ProxySettingsPolicy` API. ## Overview The `ProxySettingsPolicy` API allows Cluster Operators and Application Developers to configure the connection behavior between NGINX Gateway Fabric and upstream applications (backends). The settings in `ProxySettingsPolicy` correspond to the following NGINX directives: - [`proxy_buffering`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) - [`proxy_buffer_size`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) - [`proxy_buffers`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) - [`proxy_busy_buffers_size`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) - [`proxy_connect_timeout`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) - [`proxy_read_timeout`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) - [`proxy_send_timeout`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) `ProxySettingsPolicy` is an [Inherited PolicyAttachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that can be applied to a Gateway, HTTPRoute, or GRPCRoute in the same namespace as the `ProxySettingsPolicy`. When applied to a Gateway, the settings specified in the `ProxySettingsPolicy` affect all HTTPRoutes and GRPCRoutes attached to the Gateway. This allows Cluster Operators to set defaults for all applications using the Gateway. When applied to an HTTPRoute or GRPCRoute, the settings in the `ProxySettingsPolicy` affect only the route they are applied to. This allows Application Developers to set values for their applications based on their application's behavior or requirements. Settings applied to an HTTPRoute or GRPCRoute take precedence over settings applied to a Gateway. See the [custom policies](/ngf/overview/custom-policies.md) document for more information on policies. This guide will show you how to use the `ProxySettingsPolicy` API to configure proxy buffering for your applications. For all the possible configuration options for `ProxySettingsPolicy`, see the [API reference](/ngf/reference/api.md). ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. Create the coffee and tea example applications: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/proxy-settings-policy/app.yaml ``` The coffee application is designed to generate large responses (10KB headers and 5MB body) to demonstrate buffering requirements. The tea application returns standard responses. Create a Gateway: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/proxy-settings-policy/gateway.yaml ``` Create HTTPRoutes for the coffee and tea applications: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v/examples/proxy-settings-policy/httproutes.yaml ``` After creating the Gateway resource, NGINX Gateway Fabric will provision an NGINX Pod and Service fronting it to route traffic. Verify the gateway is created: ```shell kubectl describe gateways.gateway.networking.k8s.io gateway ``` Verify the status is `Accepted`: ```text Status: Addresses: Type: IPAddress Value: 10.96.36.219 Conditions: Last Transition Time: 2026-01-09T05:40:37Z Message: The Gateway is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Last Transition Time: 2026-01-09T05:40:37Z Message: The Gateway is programmed Observed Generation: 1 Reason: Programmed Status: True Type: Programmed ``` Save the public IP address and port(s) of the Gateway into shell variables: ```text GW_IP=XXX.YYY.ZZZ.III GW_PORT= ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Test the configuration: You can send traffic to the coffee and tea applications using the external IP address and port for the NGINX Service. Send a request to tea: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` This request should receive a response from the tea Pod: ```text Server address: 10.244.0.9:8080 Server name: tea-76c7c85bbd-cf8nz ``` Now send a request to coffee: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` This request will fail with a 502 Bad Gateway error: ```text 502 Bad Gateway

502 Bad Gateway

nginx
``` This error occurs because the coffee application generates a 10KB response header, which exceeds NGINX's default `proxy_buffer_size` (typically 4KB-8KB). You can verify this by checking the NGINX data plane logs: ```shell kubectl logs ``` Replace `` with the name of your NGINX Gateway Fabric data plane Pod (in the same namespace as your Gateway). You should see an error similar to: ```text [error] upstream sent too big header while reading response header from upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.7:8080/coffee", host: "cafe.example.com:8080" ``` This demonstrates why proper proxy buffering configuration is essential for applications that generate large response headers or bodies. ## Configure proxy buffering ### Set default proxy buffering for the Gateway To set default proxy buffering settings for the Gateway created during setup, add the following `ProxySettingsPolicy`: ```shell kubectl apply -f - < ``` You can also verify that the policy was applied to the Gateway by checking the Gateway's status: ```shell kubectl describe gateway gateway ``` Look for the `ProxySettingsPolicyAffected` condition in the Gateway status: ```text Status: Conditions: Last Transition Time: 2026-01-08T10:03:29Z Message: The ProxySettingsPolicy is applied to the resource Observed Generation: 1 Reason: PolicyAffected Status: True Type: ProxySettingsPolicyAffected ``` This condition indicates that a `ProxySettingsPolicy` has been successfully applied to the Gateway. Test the configuration by sending requests to both applications: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` The tea application should respond normally with the configured buffering settings. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` The coffee application will still fail with a 502 Bad Gateway error because the Gateway-level policy's `bufferSize: "4k"` is not large enough to handle the coffee app's 10KB response headers. We'll fix this in the next section by applying a route-specific policy. ### Set different proxy buffering for a route To set different proxy buffering settings for a particular route, you can create another `ProxySettingsPolicy` that targets the route: ```shell kubectl apply -f - <10KB) bufferSize: "16k" # Configure more and larger buffers to handle 5MB response body buffers: number: 16 size: "64k" # Set busy buffers size to allow more data to be sent to client # while still receiving from upstream busyBuffersSize: "128k" EOF ``` This `ProxySettingsPolicy` targets the coffee HTTPRoute we created in the setup by specifying it in the `targetRefs` field. It sets larger buffering values specifically for the coffee application: - `bufferSize: "16k"`: Increases the buffer size to 16KB to accommodate the coffee app's 10KB response headers - `buffers.number: 16` and `buffers.size: "64k"`: Allocates 16 buffers of 64KB each (1MB total) to efficiently handle the 5MB response body - `busyBuffersSize: "128k"`: Allows more data to be sent to the client while still receiving from the upstream Since this policy is applied to the coffee HTTPRoute, it will only affect the coffee HTTPRoute. The `ProxySettingsPolicy` we created in the previous step will continue to affect all other routes attached to the Gateway, including the tea route. Verify that the `ProxySettingsPolicy` is Accepted: ```shell kubectl describe proxysettingspolicies.gateway.nginx.org coffee-proxy-settings ``` You should see the following status: ```text Status: Ancestors: Ancestor Ref: Group: gateway.networking.k8s.io Kind: HTTPRoute Name: coffee Namespace: default Conditions: Last Transition Time: 2026-01-08T10:03:29Z Message: Policy is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Controller Name: gateway.nginx.org/nginx-gateway-controller Events: ``` Notice that the Ancestor Ref in the status is the coffee HTTPRoute instead of the Gateway. You can also verify that the policy was applied to the HTTPRoute by checking the route's status: ```shell kubectl describe httproute coffee ``` Look for the `ProxySettingsPolicyAffected` condition in the HTTPRoute status: ```text Status: Parents: Conditions: <...> Last Transition Time: 2026-01-08T10:03:29Z Message: The ProxySettingsPolicy is applied to the resource Observed Generation: 1 Reason: PolicyAffected Status: True Type: ProxySettingsPolicyAffected Controller Name: gateway.nginx.org/nginx-gateway-controller Parent Ref: Group: gateway.networking.k8s.io Kind: Gateway Name: gateway Namespace: default Section Name: http ``` Test that the policy is configured by sending requests to both applications: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` The coffee application should now successfully return the large response with the increased buffer settings. The request will complete successfully, and you'll receive the 5MB response body. ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` The tea application continues to use the Gateway-level buffering settings since no route-specific policy is applied to it. To configure a `ProxySettingsPolicy` for a GRPCRoute, you can specify the GRPCRoute in the `spec.targetRefs`: ```shell kubectl apply -f - <(ms|s|m|h)` — for example, `5s`, `500ms`, `2m`. A value with no unit is interpreted as seconds. They are fully independent: there is no required ordering between them. ### Set proxy timeouts for the Gateway To set default timeout values for all routes attached to a Gateway: ```shell kubectl apply -f - < `bufferSize` 2. **Must be less than total buffers minus one buffer**: `busyBuffersSize` < (`buffers.number` × `buffers.size`) - `buffers.size` For example, with `buffers: {number: 8, size: "4k"}` (32KB total), valid values for `busyBuffersSize` are between `bufferSize` (exclusive) and 28KB (exclusive). **Validation limitation:** NGINX Gateway Fabric validates these constraints only when all fields are set in the same policy. If you set `busyBuffersSize` in one policy and `buffers` in another (via inheritance), you are responsible for ensuring the merged configuration satisfies NGINX's requirements. If the constraints are violated, NGINX will fail to reload and log an error. ## See also - [Custom policies](/ngf/overview/custom-policies.md): learn about how NGINX Gateway Fabric custom policies work. - [API reference](/ngf/reference/api.md): all configuration fields for the `ProxySettingsPolicy` API. -------------------------------------------------------------------------------- title: "ListenerSet API" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/listener-sets/index.md -------------------------------------------------------------------------------- Learn how to use the `ListenerSet` API to specify additional listeners for a Gateway ## Overview A `ListenerSet` is a Gateway API type for specifying additional listeners for a Gateway. It decouples network listener configurations—such as ports, hostnames, and TLS termination—from the central Gateway resource and provides a mechanism to merge multiple listeners into a single Gateway. This enables multiple application developer teams who manage their own Services and Routes to configure their own listeners on a Gateway without needing to modify the Gateway itself (which may be owned by a different team). Additionally, by using ListenerSets, users are able to scale beyond the 64-listener limit of a single Gateway resource. ## Before you begin - [Install](/ngf/install/) NGINX Gateway Fabric. ## Set up Create the `coffee` application in Kubernetes by copying and pasting the following block into your terminal: ```yaml kubectl apply -f - < 80/TCP 7s ``` Create a Gateway: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. ## Configure a ListenerSet For those that are familiar with the Gateway resource, ListenerSet `spec.listeners` is a direct copy of the Gateway's `spec.listeners`. For more information, view the [API Reference](https://gateway-api.sigs.k8s.io/reference/spec/#listenerset). Create a ListenerSet: ```yaml kubectl apply -f - < ``` **Note:** In a production environment, you should have a DNS record for the external IP address that is exposed, and it should refer to the hostname that the gateway will forward for. Test the configuration: You can send traffic to the coffee and tea applications using the external IP address and port for the NGINX Service. Send a request to coffee: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` This request should receive a response from the coffee Pod: ```text Server address: 10.244.0.22:8080 Server name: coffee-654ddf664b-6mwtb ``` Send a request to tea: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea ``` This request should receive a response from the tea Pod: ```text Server address: 10.244.0.23:8080 Server name: tea-75bc9f4b6d-g9t84 ``` Before we enable rate limiting, try sending multiple requests to coffee: ```shell for i in `seq 1 10`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee; done ``` You should see all successful responses in quick succession as we have not configured any rate limiting rules yet. ## Configure Rate Limiting ### Set Rate Limiting on an HTTPRoute To set rate limit settings for the coffee HTTPRoute created during setup, add the following `RateLimitPolicy`: ```shell kubectl apply -f - < ``` You can also verify that the policy was applied to the HTTPRoute by checking the HTTPRoute's status: ```shell kubectl describe httproute coffee ``` Look for the `RateLimitPolicyAffected` condition in the HTTPRoute status: ```text Status: Conditions: Last Transition Time: 2026-01-15T22:17:46Z Message: The RateLimitPolicy is applied to the resource Observed Generation: 1 Reason: PolicyAffected Status: True Type: RateLimitPolicyAffected ``` This condition indicates that a `RateLimitPolicy` has been successfully applied to the HTTPRoute. Test the configuration by sending a request to the coffee application: ```shell curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee ``` This request should receive a response from the coffee Pod: ```text Server address: 10.244.0.22:8080 Server name: coffee-654ddf664b-6mwtb ``` When processing a single request, the rate limiting configuration has no noticeable effect. Try to exceed the set rate limit with a script that sends multiple requests. ```shell for i in `seq 1 10`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee; done ``` You should see all successful responses from the coffee Pod, but they should be spaced apart roughly one second each as expected through the rate limiting configuration. Using the same script on the tea application, we can see there are no rate limit settings applied to the tea application: ```shell for i in `seq 1 10`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea; done ``` ### Set Rate Limiting on a GRPCRoute RateLimitPolicy can also target GRPCRoutes. To do so, re-use the policy created for the coffee HTTPRoute and add an additional targetRef: ```shell kubectl apply -f - < ``` You can also verify that the policy was applied to the GRPCRoute by checking the GRPCRoute's status: ```shell kubectl describe grpcroute grpc-route ``` Look for the `RateLimitPolicyAffected` condition in the GRPCRoute status: ```text Status: Conditions: Last Transition Time: 2026-01-15T22:19:35Z Message: The RateLimitPolicy is applied to the resource Observed Generation: 1 Reason: PolicyAffected Status: True Type: RateLimitPolicyAffected ``` This condition indicates that a `RateLimitPolicy` has been successfully applied to the GRPCRoute. #### Send gRPC traffic To access the application and test the `RateLimitPolicy` has been applied to the GRPCRoute, we will use [grpcurl](https://github.com/fullstorydev/grpcurl?tab=readme-ov-file#installation]). To test our application, we will need to create a separate `.proto` source file since we are running things locally. Create a new file named `grpc.proto` with these contents: ```proto syntax = "proto3"; option go_package = "google.golang.org/grpc/examples/helloworld/helloworld"; option java_multiple_files = true; option java_package = "io.grpc.examples.helloworld"; option java_outer_classname = "HelloWorldProto"; package helloworld; // The greeting service definition. service Greeter { // Sends a greeting rpc SayHello (HelloRequest) returns (HelloReply) {} } // The request message containing the user's name. message HelloRequest { string name = 1; } // The response message containing the greetings message HelloReply { string message = 1; } ``` In the same directory, test our gRPC application through grpcurl: ```shell grpcurl -plaintext -proto grpc.proto -authority grpc.example.com -d '{"name": "exact"}' ${GW_IP}:${GW_PORT} helloworld.Greeter/SayHello ``` The request should receive a response from the gRPC backend: ```text { "message": "Hello exact" } ``` When processing a single request, the rate limiting configuration has no noticeable effect. Try to exceed the set rate limit with a script that sends multiple requests. ```shell for i in `seq 1 10`; do grpcurl -plaintext -proto grpc.proto -authority grpc.example.com -d '{"name": "exact"}' ${GW_IP}:${GW_PORT} helloworld.Greeter/SayHello; done ``` You should see all successful responses from the grpc-backend, but they should be spaced apart roughly one second each as expected through the rate limiting configuration. ### Set Rate Limiting on a Gateway Before setting rate limiting on the Gateway, verify that the tea application isn't affected by any rate limit configuration. ```shell for i in `seq 1 10`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea; done ``` You should see that these requests have successful responses with no delays. To set rate limit settings for the Gateway we created in the setup, add the following `RateLimitPolicy`: ```shell kubectl apply -f - < ``` You can also verify that the policy was applied to the Gateway by checking the Gateway's status: ```shell kubectl describe gateway gateway ``` Look for the `RateLimitPolicyAffected` condition in the Gateway status: ```text Status: Conditions: Last Transition Time: 2026-01-15T22:23:38Z Message: The RateLimitPolicy is applied to the resource Observed Generation: 1 Reason: PolicyAffected Status: True Type: RateLimitPolicyAffected ``` This condition indicates that a `RateLimitPolicy` has been successfully applied to the Gateway. Verify that the tea application is affected by the rate limit configuration: ```shell for i in `seq 1 10`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea; done ``` Even though there is no `RateLimitPolicy` specifically targeting the tea HTTPRoute, since the HTTPRoute is attached to the Gateway, it is affected by the policy. There should be a mix of successful responses alongside `503` error codes (default error code). This is expected because of the omission of the `burst` field in this `RateLimitPolicy`, causing the excessive requests to have an error code returned as the response instead of being delayed. For more information on underlying NGINX configuration of the `limit_req_module`, see the official [NGINX documentation](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html). The `RateLimitPolicy` attached to the Gateway will affect all Routes, meaning the coffee HTTPRoute and grpc-route GRPCRoute are also affected by this policy. In NGINX, a request will pass through all `limit_req` directives that apply to it, if any of the rates are exceeded, that request will be delayed and subject to the rules of that `limit_req`. In this case, since the `RateLimitPolicy` attached to the Routes has the same `key` and has a lower rate of `1r/s`, requests sent to the coffee and grpc-backend under the rate of `10r/s` will still get delayed by the `1r/s` rate. To test this, send five requests to the coffee application: ```shell for i in `seq 1 5`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee; done ``` The requests will each be delayed at a rate of one request per second. ### View and Configure Rate Limit Logs For each request that is rate limited by NGINX, a log message is associated with it. Send a few requests to both tea and coffee applications: ```shell for i in `seq 1 5`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee; done ``` ```shell for i in `seq 1 5`; do curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea; done ``` View the NGINX logs: ```shell kubectl logs ``` ```text 127.0.0.1 - - [15/Jan/2026:20:44:57 +0000] "GET /coffee HTTP/1.1" 200 161 "-" "curl/8.7.1" 2026/01/15 20:44:57 [warn] 5788#5788: *2927 delaying request, excess: 0.943, by zone "default_rl_route-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", host: "cafe.example.com:8080" 127.0.0.1 - - [15/Jan/2026:20:44:58 +0000] "GET /coffee HTTP/1.1" 200 161 "-" "curl/8.7.1" 2026/01/15 20:44:58 [info] 5788#5788: *2925 client 127.0.0.1 closed keepalive connection 2026/01/15 20:44:58 [warn] 5788#5788: *2930 delaying request, excess: 0.885, by zone "default_rl_route-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", host: "cafe.example.com:8080" 127.0.0.1 - - [15/Jan/2026:20:44:59 +0000] "GET /coffee HTTP/1.1" 200 161 "-" "curl/8.7.1" ... 127.0.0.1 - - [15/Jan/2026:20:45:05 +0000] "GET /tea HTTP/1.1" 200 155 "-" "curl/8.7.1" 127.0.0.1 - - [15/Jan/2026:20:45:05 +0000] "GET /tea HTTP/1.1" 503 190 "-" "curl/8.7.1" 2026/01/15 20:45:05 [error] 5788#5788: *2938 limiting requests, excess: 0.500 by zone "default_rl_gateway-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /tea HTTP/1.1", host: "cafe.example.com:8080" 127.0.0.1 - - [15/Jan/2026:20:45:05 +0000] "GET /tea HTTP/1.1" 200 155 "-" "curl/8.7.1" 127.0.0.1 - - [15/Jan/2026:20:45:05 +0000] "GET /tea HTTP/1.1" 503 190 "-" "curl/8.7.1" 2026/01/15 20:45:05 [error] 5788#5788: *2941 limiting requests, excess: 0.610 by zone "default_rl_gateway-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /tea HTTP/1.1", host: "cafe.example.com:8080" ... ``` You should be able to see NGINX logs at the default `error` log level showing NGINX limiting requests. Since the `coffee` application has `burst=3`, the requests are delayed and are all met with the `200` status code, while the `tea` application doesn't have `burst`, meaning all excessive requests are met with a `503`. Both the error log level and status code are adjustable in the `RateLimitPolicy`: ```shell kubectl apply -f - < ``` ```text 127.0.0.1 - - [21/Jan/2026:18:55:39 +0000] "GET /tea HTTP/1.1" 200 155 "-" "curl/8.7.1" 127.0.0.1 - - [21/Jan/2026:18:55:39 +0000] "GET /tea HTTP/1.1" 429 162 "-" "curl/8.7.1" 2026/01/21 18:55:39 [warn] 1290#1290: *595 limiting requests, excess: 0.580 by zone "default_rl_gateway-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /tea HTTP/1.1", host: "cafe.example.com:8080" 127.0.0.1 - - [21/Jan/2026:18:55:39 +0000] "GET /tea HTTP/1.1" 200 155 "-" "curl/8.7.1" 127.0.0.1 - - [21/Jan/2026:18:55:39 +0000] "GET /tea HTTP/1.1" 429 162 "-" "curl/8.7.1" 2026/01/21 18:55:39 [warn] 1292#1292: *598 limiting requests, excess: 0.470 by zone "default_rl_gateway-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /tea HTTP/1.1", host: "cafe.example.com:8080" 2026/01/21 18:55:39 [warn] 1293#1293: *599 limiting requests, excess: 0.020 by zone "default_rl_gateway-rate-limit_rule0", client: 127.0.0.1, server: cafe.example.com, request: "GET /tea HTTP/1.1", host: "cafe.example.com:8080" 127.0.0.1 - - [21/Jan/2026:18:55:39 +0000] "GET /tea HTTP/1.1" 429 162 "-" "curl/8.7.1" ``` ## Important Notes ### RateLimitPolicies set on a Gateway and Route Since Routes are affected by all `RateLimitPolicies` on a Gateway, there is no way for a `RateLimitPolicy` attached to a Route to overwrite/negate a rule set by one on a Gateway. ### Conflicts When multiple `RateLimitPolicies` select the same targetRef and specify any of dryRun, logLevel, or rejectCode, only one policy will be applied. The controller selects the policy with the highest priority (based on time created, if created at the same time, ties are calculated on alphabetical order sorting of the policy name) and rejected policies will have the `Accepted` Condition set to false with the reason `Conflicted`. ### Dry Run The [limit_req_dry_run](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_dry_run) NGINX directive can be enabled by setting `spec.rateLimit.dryRun` to `true`. In this mode, rate limit is not applied, but the number of excessive requests is accounted as usual in the shared memory zone. ### NGINX Worker process not reloading after modifying key of existing limit_req_zone In NGINX, there is a measure of protection against modifying the `key` of an existing `limit_req_zone`, if done so, this error message will show up in the NGINX logs and the worker processes won't be reloaded, meaning the new configuration won't be in effect: ```text 2026/01/21 23:45:45 [emerg] 32645#32645: limit_req (a zone name) uses the (new key name) key while previously it used the (old key name) key ``` This will occur in the `RateLimitPolicy` if the `key` for a rule is modified after creation of the policy. This won't occur upon first creation of the `RateLimitPolicy`, or when adding new rules, only when modifying an existing rule. A workaround is to either delete and re-deploy the `RateLimitPolicy` or to modify the `zoneSize` field alongside the `key`. Modifying the `zoneSize` will result in the creation of a new zone and a reloading of NGINX processes. You can verify the worker processes have been reloaded by checking for these NGINX logs: ```text ... 2026/01/22 19:17:24 [notice] 12#12: start worker process 85 2026/01/22 19:17:24 [notice] 12#12: start worker process 86 2026/01/22 19:17:24 [notice] 12#12: start worker process 87 2026/01/22 19:17:24 [notice] 48#48: gracefully shutting down 2026/01/22 19:17:24 [notice] 49#49: gracefully shutting down ... time=2026-01-22T19:17:27.978Z level=INFO msg="NGINX workers have been reloaded" correlation_id=e0c6953c-cb1d-45fd-826f-0bf66a192700 server_type=command time=2026-01-22T19:17:27.978Z level=INFO msg="NGINX reloaded" process_id=12 correlation_id=e0c6953c-cb1d-45fd-826f-0bf66a192700 server_type=command ``` ## See also - [NGINX limit_req_module](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html): for more information on the underlying NGINX directives. - [Custom policies](/ngf/overview/custom-policies.md): learn about how NGINX Gateway Fabric custom policies work. - [API reference](/ngf/reference/api.md): all configuration fields for the `RateLimitPolicy` API. -------------------------------------------------------------------------------- title: "Configure upstream HTTP/2 through Service appProtocol" url: https://docs.nginx.com/nginx-gateway-fabric/traffic-management/upstream-http2-appprotocol/index.md description: Configure NGINX Gateway Fabric to use HTTP/2 for upstream connections by setting appProtocol to kubernetes.io/h2c on a Kubernetes Service port. -------------------------------------------------------------------------------- Learn how to configure NGINX Gateway Fabric to use HTTP/2 when proxying requests to upstream services using the Service port's `appProtocol` field. ## Overview The appProtocol field on a Kubernetes Service port provides a way to specify an application protocol. Controllers such as NGINX Gateway Fabric may use this field to enable protocol-specific functionality for supported protocols. For more information, view the official [Kubernetes Service Documentation](https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol). When a Kubernetes Service port has `appProtocol` set to `kubernetes.io/h2c`, NGINX Gateway Fabric configures the corresponding NGINX location to use HTTP/2 for upstream connections by setting the [`proxy_http_version`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version) directive to `2` in the NGINX configuration. ## Important Notes - `kubernetes.io/h2c` is supported on HTTPRoutes and GRPCRoutes. It isn't supported on TLSRoutes. - When NGINX Gateway Fabric detects `kubernetes.io/h2c`, it also omits the `proxy_set_header Upgrade` and `proxy_set_header Connection` directives from the location block, because those headers are HTTP/1.1-specific and aren't used in HTTP/2 connections. ### Other supported appProtocols In addition to the `kubernetes.io/h2c` `appProtocol`, NGINX Gateway Fabric recognizes `kubernetes.io/ws` and `kubernetes.io/wss` as described in [RFC 6455](https://www.rfc-editor.org/info/rfc6455/). These `appProtocols` reference WebSocket over cleartext and WebSocket over TLS respectively and are supported natively by our default NGINX configuration. `kubernetes.io/ws` is only supported on HTTPRoutes and `kubernetes.io/wss` is only supported on TLSRoutes or on HTTPRoutes with an associated BackendTLSPolicy. If an `appProtocol` on a Service port is referenced by an unsupported Route type, that backendRef will be considered invalid and status will be written to the Route. NGINX Gateway Fabric is conformant to the information in [GEP-1911](https://gateway-api.sigs.k8s.io/geps/gep-1911/), which should be referenced for more detailed information. ## Troubleshooting - For NGINX to set `proxy_http_version 2` for a location, all valid backend references in the routing rule must have `appProtocol: kubernetes.io/h2c` set on their Service ports. If any valid backend doesn't use `kubernetes.io/h2c`, NGINX falls back to the default HTTP/1.1. ## See also - [Backend Protocol](https://gateway-api.sigs.k8s.io/guides/user-guides/backend-protocol/): the Gateway API guide for the `appProtocol` field. - [GEP-1911](https://gateway-api.sigs.k8s.io/geps/gep-1911/): for more details on how NGINX Gateway Fabric interacts with `appProtocols`. - [proxy_http_version NGINX directive](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version). -------------------------------------------------------------------------------- title: "F5 NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/index.md -------------------------------------------------------------------------------- ## About NGINX Ingress Controller is an [Ingress Controller](/glossary/#k8s-ingress-controller) implementation for NGINX that can load balance Websocket, gRPC, TCP and UDP applications. It supports standard [Ingress](/glossary/#k8s-ingress-controller) features such as content-based routing and TLS/SSL termination. Several NGINX and NGINX Plus features are available as extensions to Ingress resources through [Annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations) and the [ConfigMap](/nic/configuration/global-configuration/configmap-resource) resource. ## Featured content ### [Install NGINX Ingress Controller with Helm](/nginx-ingress-controller/install/helm) Use Helm to deploy and configure a NGINX Ingress Controller cluster ### [Migrate from Ingress-NGINX Controller](/nginx-ingress-controller/install/migrate-ingress-nginx) Replace an Ingress-NGINX cluster with NGINX Ingress Controller ### [Changelog](/nginx-ingress-controller/changelog) Review the changes from the latest NGINX Ingress Controller releases -------------------------------------------------------------------------------- title: "About" url: https://docs.nginx.com/nginx-ingress-controller/lts/overview/about/index.md -------------------------------------------------------------------------------- This document describes the F5 NGINX Ingress Controller LTS, an Ingress Controller implementation for NGINX Plus. The LTS version of NGINX Ingress Controller is based on the latest stable release of NGINX. It is *feature-frozen* at the time of release — no new features are added after release. Only critical bug fixes and security patches are delivered as patch revisions. It is ideal for users who prioritize stability and long-term support over access to the latest features. This version is supported for 36 months from the date of release, with security patches provided throughout the support period. For users who require access to the latest features and improvements, we recommend using the latest mainline release of NGINX Ingress Controller. NGINX Ingress Controller LTS is an [Ingress Controller](/glossary/#k8s-ingress-controller) implementation for [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) that can load balance Websocket, gRPC, TCP and UDP applications. NGINX Ingress Controller LTS gives you a way to manage NGINX through the [Kubernetes](https://kubernetes.io/) API, and is built to handle the continuous change that happens in Kubernetes environments. It supports standard [Ingress](/glossary/#k8s-ingress-controller) features such as content-based routing and TLS/SSL termination. NGINX Plus features are available as extensions to Ingress resources through [Annotations](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations) and the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource) resource. NGINX Ingress Controller LTS supports the [VirtualServer and VirtualServerRoute resources](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources) as alternatives to Ingress, enabling traffic splitting and advanced content-based routing. It also supports TCP, UDP and TLS Passthrough load balancing using [TransportServer resources](/nic/lts/configuration/transportserver-resource). To learn more about NGINX Ingress Controller LTS, please read the [The design of NGINX Ingress Controller LTS](/nic/lts/overview/design.md) and [Extensibility with NGINX Plus](/nic/lts/overview/nginx-plus.md) topics. -------------------------------------------------------------------------------- title: "About" url: https://docs.nginx.com/nginx-ingress-controller/overview/about/index.md -------------------------------------------------------------------------------- This document describes the F5 NGINX Ingress Controller, an Ingress Controller implementation for NGINX and NGINX Plus. NGINX Ingress Controller is an [Ingress Controller](/glossary/#k8s-ingress-controller) implementation for [NGINX](https://nginx.org) and [NGINX Plus](https://www.f5.com/products/nginx/nginx-plus) that can load balance Websocket, gRPC, TCP and UDP applications. NGINX Ingress Controller gives you a way to manage NGINX through the [Kubernetes](https://kubernetes.io/) API, and is built to handle the continuous change that happens in Kubernetes environments. It supports standard [Ingress](/glossary/#k8s-ingress-controller) features such as content-based routing and TLS/SSL termination. Several NGINX and NGINX Plus features are available as extensions to Ingress resources through [Annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations) and the [ConfigMap](/nic/configuration/global-configuration/configmap-resource) resource. NGINX Ingress Controller supports the [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources) as alternatives to Ingress, enabling traffic splitting and advanced content-based routing. It also supports TCP, UDP and TLS Passthrough load balancing using [TransportServer resources](/nic/configuration/transportserver-resource). To learn more about NGINX Ingress Controller, please read the [The design of NGINX Ingress Controller](/nic/overview/design.md) and [Extensibility with NGINX Plus](/nic/overview/nginx-plus.md) topics. -------------------------------------------------------------------------------- title: "Basic configuration" url: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/basic-configuration/index.md -------------------------------------------------------------------------------- This document shows a basic Ingress resource definition for F5 NGINX Ingress Controller. It load balances requests for two services as part of a single application. Here is a breakdown of what this Ingress resource definition means: - The `metadata.name` field defines the name of the resource `cafe‑ingress`. - The `spec.tls` field sets up SSL/TLS termination: - The `hosts` field applies the certificate and key to the `cafe.example.com` host. - The `secretName` references a secret resource by its name, `cafe‑secret`. The secret must belong to the same namespace as the Ingress, of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that hold the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls>). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the hosts to which the secret is applied. - The `spec.rules` field defines a host with the domain name `cafe.example.com`. - The `paths` field defines two path‑based rules: - The rule with the path `/tea` instructs NGINX to distribute the requests with the `/tea` URI among the pods of the *tea* service, which is deployed with the name `tea‑svc` in the cluster. - The rule with the path `/coffee` instructs NGINX to distribute the requests with the `/coffee` URI among the pods of the *coffee* service, which is deployed with the name `coffee‑svc` in the cluster. - Both rules instruct NGINX to distribute the requests to `port 80` of the corresponding service (the `servicePort` field). To learn more about the Ingress resource, view [the official Kubernetes documentation for Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/). **Note:** For complete instructions on deploying Ingress and Secret resources in the cluster, see the [complete example](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/complete-example) in the GitHub repository. ## New features available in Kubernetes 1.18 Starting from Kubernetes 1.18, you can use the following new features: - The host field supports wildcard domain names, such as `*.example.com`. - The path supports different matching rules with the new field `pathType`, which takes the following values: `Prefix` for prefix-based matching, `Exact` for exact matching and `ImplementationSpecific`, which is the default type and is the same as `Prefix`. For example: ```yaml {hl_lines=[2, 7, 14]} - path: /tea pathType: Prefix backend: serviceName: tea-svc servicePort: 80 - path: /tea/green pathType: Exact backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80 ``` - The `ingressClassName` field is now supported: ```yaml {hl_lines=[6]} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress spec: ingressClassName: nginx tls: - hosts: - cafe.example.com secretName: cafe-secret rules: - host: cafe.example.com . . . ``` When using this field you need to create the `IngressClass` resource with the corresponding `name`. View the [Create common resources](/nic/install/manifests.md#create-common-resources) section of the Installation with Manifests topic for more information. ## Restrictions NGINX Ingress Controller imposes the following restrictions on Ingress resources: - The `host` field is required by default. To allow Ingress resources to omit the `host` field, use the [`-allow-empty-ingress-host`](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-allow-empty-ingress-host) command-line argument. - When `-allow-empty-ingress-host` is enabled, `spec.tls` is not supported on Ingress resources without a `host`. To configure TLS, use the [`-default-server-tls-secret`](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-default-server-tls-secret) command-line argument instead. - The `host` value needs to be unique among all Ingress and VirtualServer resources unless the Ingress resource is a [mergeable minion](/nic/configuration/ingress-resources/cross-namespace-configuration.md). View the [Host and Listener collisions](/nic/configuration/host-and-listener-collisions.md) topic for more information. - The `path` field in `spec.rules[].http.paths[]` is required for `Exact` and `Prefix` `pathTypes`. - The ImplementationSpecific `pathType` is treated as equivalent to `Prefix` `pathType`, with the exception that when this `pathType` is configured, the `path` field in `spec.rules[].http.paths[]` is not mandatory. `path` defaults to `/` if not set but the `pathType` is set to ImplementationSpecific. ## Advanced configuration NGINX Ingress Controller generates NGINX configuration by executing a template file that contains the configuration options. These options are set with the Ingress resource and NGINX Ingress Controller's ConfigMap. The Ingress resource only allows you to use basic NGINX features: host and path-based routing and TLS termination. For advanced configuration, you have two options: - [Annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md) can be used to rewrite request URIs or inserting additional response headers. - [Snippets](/nic/configuration/ingress-resources/advanced-configuration-with-snippets) can be used to insert raw NGINX configuration, changing generated files. Additionally, it is possible to customize the template, described in the [Custom templates](/nic/configuration/global-configuration/custom-templates.md) topic. -------------------------------------------------------------------------------- title: "Basic configuration" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/ingress-resources/basic-configuration/index.md -------------------------------------------------------------------------------- This document shows a basic Ingress resource definition for F5 NGINX Ingress Controller LTS. It load balances requests for two services as part of a single application. Here is a breakdown of what this Ingress resource definition means: - The `metadata.name` field defines the name of the resource `cafe‑ingress`. - The `spec.tls` field sets up SSL/TLS termination: - The `hosts` field applies the certificate and key to the `cafe.example.com` host. - The `secretName` references a secret resource by its name, `cafe‑secret`. The secret must belong to the same namespace as the Ingress, of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that hold the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls>). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the hosts to which the secret is applied. - The `spec.rules` field defines a host with the domain name `cafe.example.com`. - The `paths` field defines two path‑based rules: - The rule with the path `/tea` instructs NGINX to distribute the requests with the `/tea` URI among the pods of the *tea* service, which is deployed with the name `tea‑svc` in the cluster. - The rule with the path `/coffee` instructs NGINX to distribute the requests with the `/coffee` URI among the pods of the *coffee* service, which is deployed with the name `coffee‑svc` in the cluster. - Both rules instruct NGINX to distribute the requests to `port 80` of the corresponding service (the `servicePort` field). To learn more about the Ingress resource, view [the official Kubernetes documentation for Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/). **note:** For complete instructions on deploying Ingress and Secret resources in the cluster, see the [complete example](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/complete-example) in the GitHub repository. ## New features available in Kubernetes 1.18 Starting from Kubernetes 1.18, you can use the following new features: - The host field supports wildcard domain names, such as `*.example.com`. - The path supports different matching rules with the new field `pathType`, which takes the following values: `Prefix` for prefix-based matching, `Exact` for exact matching and `ImplementationSpecific`, which is the default type and is the same as `Prefix`. For example: ```yaml {hl_lines=[2, 7, 14]} - path: /tea pathType: Prefix backend: serviceName: tea-svc servicePort: 80 - path: /tea/green pathType: Exact backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80 ``` - The `ingressClassName` field is now supported: ```yaml {hl_lines=[6]} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress spec: ingressClassName: nginx tls: - hosts: - cafe.example.com secretName: cafe-secret rules: - host: cafe.example.com . . . ``` When using this field you need to create the `IngressClass` resource with the corresponding `name`. View the [Create common resources](/nic/lts/install/manifests.md#create-common-resources) section of the Installation with Manifests topic for more information. ## Restrictions NGINX Ingress Controller LTS imposes the following restrictions on Ingress resources: - When defining an Ingress resource, the `host` field is required. - The `host` value needs to be unique among all Ingress and VirtualServer resources unless the Ingress resource is a [mergeable minion](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md). View the [Host and Listener collisions](/nic/lts/configuration/host-and-listener-collisions.md) topic for more information. - The `path` field in `spec.rules[].http.paths[]` is required for `Exact` and `Prefix` `pathTypes`. - The ImplementationSpecific `pathType` is treated as equivalent to `Prefix` `pathType`, with the exception that when this `pathType` is configured, the `path` field in `spec.rules[].http.paths[]` is not mandatory. `path` defaults to `/` if not set but the `pathType` is set to ImplementationSpecific. ## Advanced configuration NGINX Ingress Controller LTS generates NGINX configuration by executing a template file that contains the configuration options. These options are set with the Ingress resource and NGINX Ingress Controller LTS's ConfigMap. The Ingress resource only allows you to use basic NGINX features: host and path-based routing and TLS termination. For advanced configuration, you have two options: - [Annotations](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations.md) can be used to rewrite request URIs or inserting additional response headers. - [Snippets](/nic/lts/configuration/ingress-resources/advanced-configuration-with-snippets) can be used to insert raw NGINX configuration, changing generated files. Additionally, it is possible to customize the template, described in the [Custom templates](/nic/lts/configuration/global-configuration/custom-templates.md) topic. -------------------------------------------------------------------------------- title: "Before you begin" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/before-you-begin/index.md -------------------------------------------------------------------------------- Use a Policy resource to define reusable configuration, such as access control, CORS, egress mTLS, and WAF. Attach that configuration to [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) or [Ingress resources](/nic/configuration/ingress-resources). A Policy resource is a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). A Policy resource works together with [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) and [Ingress resources](/nic/configuration/ingress-resources). Create these resources separately before you attach a policy to them. ## What's next Learn about the [policy specification](/nic/configuration/policy-resource/policy-specification.md) and the supported policy types. -------------------------------------------------------------------------------- title: "Before you begin" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/policy-resource/before-you-begin/index.md -------------------------------------------------------------------------------- Use a Policy resource to configure features like access control and rate limiting. Add this configuration to your [VirtualServer, VirtualServerRoute resources](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md) and [Ingress resources](/nic/lts/configuration/ingress-resources). A Policy resource is a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). This section is reference documentation for the Policy resource. For an example Policy for access control, see the [GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v/examples/custom-resources/access-control). A Policy resource works together with [VirtualServer, VirtualServerRoute resources](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md) and [Ingress resources](/nic/lts/configuration/ingress-resources). Create these resources separately. ## What's next Learn about the [policy specification](/nic/lts/configuration/policy-resource/policy-specification.md). -------------------------------------------------------------------------------- title: "Build NGINX Ingress Controller with F5 DoS for NGINX" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-dos/installation/index.md -------------------------------------------------------------------------------- This document explains how to build an image for F5 NGINX Ingress Controller with F5 DoS for NGINX from source code. **Note:** If you'd rather not build your own NGINX Ingress Controller image, see the [pre-built image options](#pre-built-images) at the end of this guide. ## Before you start - To use F5 DoS for NGINX with NGINX Ingress Controller, you must have NGINX Plus. ## Prepare the environment {#prepare-environment} Get your system ready for building and pushing the NGINX Ingress Controller image with F5 DoS for NGINX. 1. Sign in to your private registry. Replace `` with the path to your own private registry. ```shell docker login ``` 2. Clone the NGINX Ingress Controller GitHub repository. Replace `` with the version of NGINX Ingress Controller you want. ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch cd kubernetes-ingress ``` For instance if you want to clone version v, the commands to run would be: ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch v cd kubernetes-ingress/deployments ``` ## Build the image {#build-docker-image} Follow these steps to build the NGINX Controller Image with F5 DoS for NGINX. 1. Place your NGINX Plus license files (_nginx-repo.crt_ and _nginx-repo.key_) in the project's root folder. To verify they're in place, run: ```shell ls nginx-repo.* ``` You should see: ```shell nginx-repo.crt nginx-repo.key ``` 2. Build the image. Replace `` with your chosen build option and `` with your private registry's path. Refer to the [Makefile targets](#makefile-targets) table below for the list of build options. ```shell make PREFIX=/nginx-plus-ingress TARGET=download ``` For example, to build a Debian-based image with NGINX Plus and F5 DoS for NGINX, run: ```shell make debian-image-dos-plus PREFIX=/nginx-plus-ingress TARGET=download ``` **What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](/nic/install/build.md#makefile-details). This version number is used for tracking and deployment purposes. **Note:** In the event a patch version of NGINX Plus is released, make sure to rebuild your image to get the latest version. If your system is caching the Docker layers and not updating the packages, add `DOCKER_BUILD_OPTIONS="--pull --no-cache"` to the make command. ### Makefile targets | Makefile Target | Description | Compatible Systems | |---------------------------|-------------------------------------------------------------------|---------------------| | **debian-image-dos-plus** | Builds a Debian-based image with NGINX Plus and the [F5 DoS for NGINX](/nginx-app-protect-dos/) module. | Debian | | **debian-image-nap-dos-plus** | Builds a Debian-based image with NGINX Plus, [F5 DoS for NGINX](/nginx-app-protect-dos/), and [F5 WAF for NGINX](/nginx-app-protect/). | Debian | | **ubi-image-dos-plus** | Builds a UBI-based image with NGINX Plus and the [F5 DoS for NGINX](/nginx-app-protect-dos/) module. | OpenShift | | **ubi-image-nap-dos-plus** | Builds a UBI-based image with NGINX Plus, [F5 DoS for NGINX](/nginx-app-protect-dos/), and [F5 WAF for NGINX](/nginx-app-protect/). | OpenShift | **Note:** For the complete list of _Makefile_ targets and customizable variables, see the [Build NGINX Ingress Controller](/nic/install/build.md#makefile-details) topic. ## Push the image to your private registry Once you've successfully built the NGINX Ingress Controller image with F5 DoS for NGINX, the next step is to upload it to your private Docker registry. This makes the image available for deployment to your Kubernetes cluster. To upload the image, run the following command. If you're using a custom tag, add `TAG=your-tag` to the end of the command. Replace `` with your private registry's path. ```shell make push PREFIX=/nginx-plus-ingress ``` ## Set up role-based access control (RBAC) {#set-up-rbac} **Note:** To complete these steps you need admin access to your cluster. Refer to your Kubernetes platform's documentation to set up admin access. For Google Kubernetes Engine (GKE), you can refer to their [Role-Based Access Control guide](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control). 1. Create a namespace and a service account: ```shell kubectl apply -f deployments/common/ns-and-sa.yaml ``` 2. Create a cluster role and binding for the service account: ```shell kubectl apply -f deployments/rbac/rbac.yaml ```
If you're planning to use F5 WAF for NGINX or F5 DoS for NGINX, additional roles and bindings are needed. 1. (F5 WAF for NGINX only) Create the *App Protect* role and binding: ```shell kubectl apply -f deployments/rbac/ap-rbac.yaml ``` 2. (F5 DoS for NGINX only) Create the *App Protect DoS* role and binding: ```shell kubectl apply -f deployments/rbac/apdos-rbac.yaml ``` ## Create common resources {#create-common-resources} In this section, you'll create resources that most NGINX Ingress Controller installations require: 1. (Optional) Create a secret for the default NGINX server's TLS certificate and key. Complete this step only if you're using the [default server TLS secret](/nic/configuration/global-configuration/command-line-arguments#cmdoption-default-server-tls-secret.md) command-line argument. If you're not, feel free to skip this step. By default, the server returns a _404 Not Found_ page for all requests when no ingress rules are set up. Although we provide a self-signed certificate and key for testing purposes, we recommend using your own certificate. ```shell make secrets kubectl apply -f examples/shared-examples/default-server-secret/default-server-secret.yaml ``` 2. Create a ConfigMap to customize your NGINX settings: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` 3. Create an `IngressClass` resource. NGINX Ingress Controller won't start without an `IngressClass` resource. ```shell kubectl apply -f deployments/common/ingress-class.yaml ``` If you want to make this NGINX Ingress Controller instance your cluster's default, uncomment the `ingressclass.kubernetes.io/is-default-class` annotation. This action will auto-assign `IngressClass` to new ingresses that don't specify an `ingressClassName`. ## Create custom resources {#create-custom-resources} To make sure your NGINX Ingress Controller pods reach the `Ready` state, you'll need to create custom resource definitions (CRDs) for various components. Alternatively, you can disable this requirement by setting the `-enable-custom-resources` command-line argument to `false`. There are two ways you can install the custom resource definitions: 1. Using a URL to apply a single CRD yaml file, which we recommend. 1. Applying your local copy of the CRD yaml files, which requires you to clone the repository. The core custom CRDs are the following: - [VirtualServer and VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) - [TransportServer](/nic/configuration/transportserver-resource.md) - [Policy](/nic/configuration/policy-resource/before-you-begin.md) - [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) #### Install CRDs from single YAML ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` #### Install CRDs after cloning the repo **Note:** Read the steps outlined in [Upgrade from 3.x to 4.x](/nic/install/upgrade.md#upgrade-from-3x-to-4x) before running the CRD upgrade and perform the steps if applicable. ```shell kubectl apply -f config/crd/bases/k8s.nginx.org_virtualservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_virtualserverroutes.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_transportservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_policies.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_globalconfigurations.yaml ``` ## Create App Protect DoS custom resources #### Install CRDs from single YAML This single YAML file creates CRDs for the following resources: - `APDosPolicy` - `APDosLogConf` - `DosProtectedResource` ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds-nap-dos.yaml ``` #### Install CRDs after cloning the repo These YAML files create CRDs for the following resources: - `APDosPolicy` - `APDosLogConf` - `DosProtectedResource` ```shell kubectl apply -f config/crd/bases/appprotectdos.f5.com_apdoslogconfs.yaml kubectl apply -f config/crd/bases/appprotectdos.f5.com_apdospolicy.yaml kubectl apply -f config/crd/bases/appprotectdos.f5.com_dosprotectedresources.yaml ``` ## Deploy NGINX Ingress Controller {#deploy-ingress-controller} You have two options for deploying NGINX Ingress Controller: - **Deployment**. Choose this method for the flexibility to dynamically change the number of NGINX Ingress Controller replicas. - **DaemonSet**. Choose this method if you want NGINX Ingress Controller to run on all nodes or a subset of nodes. - **StatefulSet**. Choose this method when you need stable, persistent storage and ordered deployment/scaling for your NGINX Ingress Controller pods. Before you start, update the [command-line arguments](/nic/configuration/global-configuration/command-line-arguments.md) for the NGINX Ingress Controller container in the relevant manifest file to meet your specific requirements. ### Using a Deployment For additional context on managing containers using Kubernetes Deployments, refer to the official Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) documentation. When you deploy NGINX Ingress Controller as a Deployment, Kubernetes automatically sets up a single NGINX Ingress Controller pod. - For NGINX, run: ```shell kubectl apply -f deployments/deployment/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/deployment/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a DaemonSet For additional context on managing containers using Kubernetes DaemonSets, refer to the official Kubernetes [DaemonSets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) documentation. When you deploy NGINX Ingress Controller as a DaemonSet, Kubernetes creates an Ingress Controller pod on every node in the cluster. - For NGINX, run: ```shell kubectl apply -f deployments/daemon-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/daemon-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a StatefulSet For additional context on managing containers using Kubernetes StatefulSets, refer to the official Kubernetes [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) documentation. When you deploy NGINX Ingress Controller as a StatefulSet, Kubernetes creates pods with stable network identities and persistent storage. - For NGINX, run: ```shell kubectl apply -f deployments/stateful-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/stateful-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. **Note:** StatefulSets include persistent volume claims for nginx cache storage via `volumeClaimTemplates`. You may need to configure a StorageClass in your cluster or modify the volumeClaimTemplates section in the manifest to match your storage requirements. Other volumes (like those needed for App Protect modules) are configured in the regular `volumes` section, not in volumeClaimTemplates. ## Install the App Protect DoS Arbitrator **Note:** If you install multiple NGINX Ingress Controllers in the same namespace, they will need to share the same Arbitrator because there can only be one Arbitrator in a single namespace. ### Helm Chart The App Protect DoS Arbitrator can be installed using the [F5 DoS for NGINX Helm Chart](https://github.com/nginxinc/nap-dos-arbitrator-helm-chart). If you have the NGINX Helm Repository already added, you can install the App Protect DoS Arbitrator by running the following command: ```shell helm install my-release-dos nginx-stable/nginx-appprotect-dos-arbitrator ``` ### YAML Manifests Alternatively, you can install the App Protect DoS Arbitrator using the YAML manifests provided in the NGINX Ingress Controller repo. 1. Create the namespace and service account: ```shell kubectl apply -f common/ns-and-sa.yaml ``` 2. Deploy the F5 WAF for NGINX Arbitrator as a Deployment and service: ```shell kubectl apply -f deployment/appprotect-dos-arb.yaml kubectl apply -f service/appprotect-dos-arb-svc.yaml ``` ## Enable F5 DoS for NGINX module To enable the F5 DoS for NGINX Module: - Add the `enable-app-protect-dos` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-app-protect-dos) to your Deployment, DaemonSet, or StatefulSet file. ## Confirm NGINX Ingress Controller is running To confirm the NGINX Ingress Controller pods are operational, run: ```shell kubectl get pods --namespace=nginx-ingress ``` For more information, see the [Configuration guide](/nic/integrations/app-protect-dos/configuration.md),the [NGINX Ingress Controller with App Protect DoS example for VirtualServer](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/app-protect-dos) and the [NGINX Ingress Controller with App Protect DoS example for Ingress](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-dos). ## Alternatives to building your own image {#pre-built-images} If you prefer not to build your own NGINX Ingress Controller image, you can use pre-built images. Here are your options: - Download the image using your NGINX Ingress Controller subscription certificate and key. View the [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) topic. - The [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) topic describes how to use your subscription JWT token to get the image. -------------------------------------------------------------------------------- title: "Build NGINX Ingress Controller with F5 WAF for NGINX" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-waf-v5/installation/index.md -------------------------------------------------------------------------------- This document explains how to build a F5 NGINX Ingress Controller image with F5 WAF for NGINX v5 from source code. **Note:** If you'd rather not build your own NGINX Ingress Controller image, see the [pre-built image options](#pre-built-images) at the end of this guide. ## Before you begin - To use F5 WAF for NGINX with NGINX Ingress Controller, you must have NGINX Plus. NGINX Ingress Controller supports the following versions of [F5 WAF for NGINX](https://docs.nginx.com/waf/): | NIC Version | NAP-WAF Version | Config Manager | Enforcer | | ------------------- | --------------- | -------------- | -------- | | | 37.1+ | | | | 5.5.4 | 37.0+5.690.0 | 5.14.0 | 5.14.0 | | 5.4.3 | 37.0+5.635.1 | 5.13.1 | 5.13.1 | | 5.3.4 | 35+5.527.0 | 5.11.2 | 5.11.2 | | 5.2.1 | 35+5.527.0 | 5.9.0 | 5.9.0 | | 5.1.1 | 35+5.498 | 5.8.0 | 5.8.0 | | 5.0.0 | 34+5.342 | 5.6.0 | 5.6.0 | | 4.0.1 | 33+5.264 | 5.5.0 | 5.5.0 | | 3.7.2 | 32+5.1 | 5.3.0 | 5.3.0 | | 3.6.2 | 32+5.48 | 5.2.0 | 5.2.0 | ## Prepare the environment Get your system ready for building and pushing the NGINX Ingress Controller image with F5 WAF for NGINX v5. 1. Sign in to your private registry. Replace `` with the path to your own private registry. ```shell docker login ``` 1. Pull the WAF Config Manager image: ```shell docker pull private-registry.nginx.com/nap/waf-config-mgr: ``` 1. Pull the WAF Enforcer Docker image ```shell docker pull private-registry.nginx.com/nap/waf-enforcer: ``` 1. Pull the WAF IP Intelligence image (if you plan to use the WAF IP Intelligence module with your WAF v5 policies): ```shell docker pull private-registry.nginx.com/nap/waf-ip-intelligence: ``` 1. Clone the NGINX Ingress Controller repository: ```console git clone https://github.com/nginx/kubernetes-ingress.git --branch v cd kubernetes-ingress ``` ## Build the image Follow these steps to build the NGINX Controller Image with F5 WAF for NGINX v5. 1. Place your NGINX Plus license files (_nginx-repo.crt_ and _nginx-repo.key_) in the project's root folder. To verify they're in place, run: ```shell ls nginx-repo.* ``` You should see: ```shell nginx-repo.crt nginx-repo.key ``` 2. Build the image. Replace `` with your chosen build option and `` with your private registry's path. Refer to the [Makefile targets](#makefile-targets) table below for the list of build options. ```shell make PREFIX=/nginx-plus-ingress TARGET=download ``` For example, to build a Debian-based image with NGINX Plus and F5 WAF for NGINX v5, run: ```shell make debian-image-nap-v5-plus PREFIX=/nginx-plus-ingress TARGET=download ``` **What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](/nic/install/build.md#makefile-details). This version number is used for tracking and deployment purposes. **Note:** In the event a patch of NGINX Plus is released, make sure to rebuild your image to get the latest version. If your system is caching the Docker layers and not updating the packages, add `DOCKER_BUILD_OPTIONS="--pull --no-cache"` to the make command. ### Makefile targets {#makefile-targets} Create Docker image for NGINX Ingress Controller (Alpine with NGINX Plus, F5 WAF for NGINX v5 and FIPS) | Makefile Target | Description | Compatible Systems | |---------------------------|-------------------------------------------------------------------|---------------------| | **alpine-image-nap-v5-plus-fips** | Builds a Alpine-based image with NGINX Plus and the [F5 WAF for NGINX v5](/nginx-app-protect-waf/v5/) module with FIPS. | Alpine | | **debian-image-nap-v5-plus** | Builds a Debian-based image with NGINX Plus and the [F5 WAF for NGINX v5](/nginx-app-protect-waf/v5/) module. | Debian | | **ubi-image-nap-v5-plus** | Builds a UBI-based image with NGINX Plus and the [F5 WAF for NGINX v5](/nginx-app-protect-waf/v5/) module. | OpenShift | | **ubi-image-nap-dos-v5-plus** | Builds a UBI-based image with NGINX Plus, [F5 WAF for NGINX v5](/nginx-app-protect-waf/v5/), and [F5 DoS for NGINX](/nginx-app-protect-dos/). | OpenShift | **Note:** For the complete list of _Makefile_ targets and customizable variables, see the [Build NGINX Ingress Controller](/nic/install/build.md#makefile-details) guide. If you intend to use [external references](/nginx-app-protect-waf/v5/configuration-guide/configuration/#external-references) in F5 WAF for NGINX policies, you may want to provide a custom CA certificate to authenticate with the hosting server. To do so, place the `*.crt` file in the build folder and uncomment the lines following this comment: `#Uncomment the lines below if you want to install a custom CA certificate` **Note:** External references are deprecated in NGINX Ingress Controller and will not be supported in future releases. ## Push the images to your private registry Once you've successfully pulled the WAF v5 manager and enforcer images and built the NGINX Ingress Controller image with F5 WAF for NGINX v5, the next step is to upload them to your private Docker registry. This makes the image available for deployment to your Kubernetes cluster. To upload the image, run the following command. If you're using a custom tag, add `TAG=your-tag` to the end of the command. Replace `` with your private registry's path. ```shell make push PREFIX=/nginx-plus-ingress ``` To upload the WAF config manager and enforcer images run the following commands: ```shell docker push /waf-config-mgr: ``` ```shell docker push /waf-enforcer: ``` To make sure your NGINX Ingress Controller pods reach the `Ready` state, you'll need to create custom resource definitions (CRDs) for various components. Alternatively, you can disable this requirement by setting the `-enable-custom-resources` command-line argument to `false`. There are two ways you can install the custom resource definitions: 1. Using a URL to apply a single CRD yaml file, which we recommend. 1. Applying your local copy of the CRD yaml files, which requires you to clone the repository. The core custom CRDs are the following: - [VirtualServer and VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) - [TransportServer](/nic/configuration/transportserver-resource.md) - [Policy](/nic/configuration/policy-resource/before-you-begin.md) - [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) #### Install CRDs from single YAML ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` #### Install CRDs after cloning the repo **Note:** Read the steps outlined in [Upgrade from 3.x to 4.x](/nic/install/upgrade.md#upgrade-from-3x-to-4x) before running the CRD upgrade and perform the steps if applicable. ```shell kubectl apply -f config/crd/bases/k8s.nginx.org_virtualservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_virtualserverroutes.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_transportservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_policies.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_globalconfigurations.yaml ``` ## Deploy NGINX Ingress Controller {#deploy-ingress-controller} **important:** NGINX Ingress Controller with the AppProtect WAF v5 module works only with policy bundles. You need to modify the Deployment, DaemonSet, or StatefulSet file to include volumes, volume mounts and two WAF 5 docker images: `waf-config-mgr` and `waf-enforcer`. NGINX Ingress Controller **requires** the volume mount path to be `/etc/app_protect/bundles`. #### With Helm Below are examples of a `PersistentVolume` and `PersistentVolumeClaim` that you can reference in your Helm values: ```yaml ... volumes: - name: persistentVolumeClaim: claimName: ... ``` Add volume mounts to the `containers` section: ```yaml ... volumeMounts: - name: mountPath: /etc/app_protect/bundles ... ``` ### Enabling WAF v5 Start by setting `controller.appprotect.enable` to `true` in your Helm values. This will the standard F5 WAF for NGINX features. Afterwords, set `controller.approtect.v5` to `true`. This ensures that both the `waf-enforcer` and `waf-config-mgr` containers are deployed alongside the NGINX Ingress Controller containers. These two additional containers are required when using F5 WAF for NGINX v5. Your Helm values should look something like this: ```yaml controller: ... ## Support for F5 WAF for NGINX appprotect: ## Enable the F5 WAF for NGINX module in the Ingress Controller. enable: true ## Enables F5 WAF for NGINX v5. v5: true ``` ### Configuring volumes Whether you have created a new `PersistentVolume` and `PersistentVolumeClaim`, or you are referencing an existing `PersistentVolumeClaim`, update the `app-protect-bundles` volume to reference your `PersistentVolumeClaim`. Example helm values: ```yaml ... controller: ... appprotect: ... volumes: - name: app-protect-bundles persistentVolumeClaim: claimName: ... ``` **Note:** By default, `emptyDir` mounts are used. Bundles that are added to these kind of volume mounts will **NOT** persist across pod restarts. Example default volumes: ```yaml ... controller: ... appprotect: ... volumes: - name: app-protect-bundles emptyDir: {} ... ``` ### Configuring `readOnlyRootFilesystem` Create required volumes: ```yaml volumes: - name: nginx-etc emptyDir: {} - name: nginx-cache # do not set this value in statefulset if volumeclaimtemplate is set emptyDir: {} # do not set this value in statefulset if volumeclaimtemplate is set - name: nginx-lib emptyDir: {} - name: nginx-log emptyDir: {} - name: app-protect-bd-config emptyDir: {} - name: app-protect-config emptyDir: {} - name: app-protect-bundles emptyDir: {} ``` Set `controller.securityContext.readOnlyRootFilesystem` to `true`. Example Helm values: ```yaml controller: ... securityContext: readOnlyRootFilesystem: true ... ``` Set `controller.appprotect.enforcer.securityContext.readOnlyRootFilesystem` to `true`. Example Helm values: ```yaml controller: ... appprotect: ... enforcer: securityContext: readOnlyRootFilesystem: true ... ``` Set `controller.appprotect.configManager.securityContext.readOnlyRootFilesystem` to `true`. Example Helm values: ```yaml controller: ... appprotect: ... configManager: securityContext: readOnlyRootFilesystem: true ... ``` #### With Manifest You have two options for deploying NGINX Ingress Controller: - **Deployment**. Choose this method for the flexibility to dynamically change the number of NGINX Ingress Controller replicas. - **DaemonSet**. Choose this method if you want NGINX Ingress Controller to run on all nodes or a subset of nodes. - **StatefulSet**. Choose this method when you need stable, persistent storage and ordered deployment/scaling for your NGINX Ingress Controller pods. ### Set up role-based access control (RBAC) {#set-up-rbac} **Note:** To complete these steps you need admin access to your cluster. Refer to your Kubernetes platform's documentation to set up admin access. For Google Kubernetes Engine (GKE), you can refer to their [Role-Based Access Control guide](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control). 1. Create a namespace and a service account: ```shell kubectl apply -f deployments/common/ns-and-sa.yaml ``` 2. Create a cluster role and binding for the service account: ```shell kubectl apply -f deployments/rbac/rbac.yaml ``` ### Volumes and VolumeMounts Add a `volumes` section to deployment template spec: ```yaml ... volumes: - name: persistentVolumeClaim: claimName: ... ``` Add volume mounts to the `containers` section: ```yaml ... volumeMounts: - name: mountPath: /etc/app_protect/bundles ... ``` ### WAF Config Manager and WAF Enforcer Add `waf-config-mgr` image to the `containers` section: ```yaml ... - name: waf-config-mgr image: private-registry.nginx.com/nap/waf-config-mgr: imagePullPolicy: IfNotPresent securityContext: allowPrivilegeEscalation: false capabilities: drop: - all volumeMounts: - name: app-protect-bd-config mountPath: /opt/app_protect/bd_config - name: app-protect-config mountPath: /opt/app_protect/config - name: app-protect-bundles mountPath: /etc/app_protect/bundles ... ``` Add `waf-enforcer` image to the `containers` section: ```yaml ... - name: waf-enforcer image: private-registry.nginx.com/nap/waf-enforcer: imagePullPolicy: IfNotPresent env: - name: ENFORCER_PORT value: "50000" - name: ENFORCER_CONFIG_TIMEOUT value: "0" volumeMounts: - name: app-protect-bd-config mountPath: /opt/app_protect/bd_config ... ``` ### Update NIC container in deployment, daemonset, or statefulset Add `volumeMounts` as below: ```yaml ... - image: : imagePullPolicy: IfNotPresent name: nginx-plus-ingress volumeMounts: - name: app-protect-bd-config mountPath: /opt/app_protect/bd_config - name: app-protect-config mountPath: /opt/app_protect/config - name: app-protect-bundles mountPath: /etc/app_protect/bundles ... ``` ### Configure `readOnlyRootFilesystem` Add `readOnlyRootFilesystem` to the NIC container and set valut to `true` as below: ```yaml ... - image: : imagePullPolicy: IfNotPresent name: nginx-plus-ingress ... securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 101 readOnlyRootFilesystem: true ... volumeMounts: - mountPath: /etc/nginx name: nginx-etc - mountPath: /var/cache/nginx name: nginx-cache - mountPath: /var/lib/nginx name: nginx-lib - mountPath: /var/log/nginx name: nginx-log - mountPath: /opt/app_protect/bd_config name: app-protect-bd-config - mountPath: /opt/app_protect/config name: app-protect-config - mountPath: /etc/app_protect/bundles name: app-protect-bundles ... ``` Add `readOnlyRootFilesystem` to the `waf-config-mgr` container and set value to `true` as below: ```yaml ... - name: waf-config-mgr image: private-registry.nginx.com/nap/waf-config-mgr: imagePullPolicy: IfNotPresent ... securityContext: readOnlyRootFilesystem: true ... ... ``` Add `readOnlyRootFilesystem` to the `waf-enforcer` container and set value to `true` as below: ```yaml ... - name: waf-enforcer image: private-registry.nginx.com/nap/waf-enforcer: imagePullPolicy: IfNotPresent ... securityContext: readOnlyRootFilesystem: true ... ... ``` **Note:** **StatefulSet Volume Configuration**: When using StatefulSet deployments, the `nginx-cache` volume is automatically provided via `volumeClaimTemplates` for persistent storage. F5 WAF for NGINX v5 volumes (like app-protect-config, app-protect-bundles) are still configured as regular volumes in the `volumes` section. Use `emptyDir` for temporary data or PersistentVolumeClaims if you need persistence for App Protect configurations across pod restarts. ### Using a Deployment For additional context on managing containers using Kubernetes Deployments, refer to the official Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) documentation. When you deploy NGINX Ingress Controller as a Deployment, Kubernetes automatically sets up a single NGINX Ingress Controller pod. - For NGINX, run: ```shell kubectl apply -f deployments/deployment/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/deployment/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a DaemonSet For additional context on managing containers using Kubernetes DaemonSets, refer to the official Kubernetes [DaemonSets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) documentation. When you deploy NGINX Ingress Controller as a DaemonSet, Kubernetes creates an Ingress Controller pod on every node in the cluster. - For NGINX, run: ```shell kubectl apply -f deployments/daemon-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/daemon-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a StatefulSet For additional context on managing containers using Kubernetes StatefulSets, refer to the official Kubernetes [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) documentation. When you deploy NGINX Ingress Controller as a StatefulSet, Kubernetes creates pods with stable network identities and persistent storage. - For NGINX, run: ```shell kubectl apply -f deployments/stateful-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/stateful-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. **Note:** StatefulSets include persistent volume claims for nginx cache storage via `volumeClaimTemplates`. You may need to configure a StorageClass in your cluster or modify the volumeClaimTemplates section in the manifest to match your storage requirements. Other volumes (like those needed for App Protect modules) are configured in the regular `volumes` section, not in volumeClaimTemplates. ### Enable F5 WAF for NGINX module To enable the F5 DoS for NGINX Module: - Add the `enable-app-protect` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-app-protect) to your Deployment, DaemonSet, or StatefulSet file. ## Confirm NGINX Ingress Controller is running To confirm the NGINX Ingress Controller pods are operational, run: ```shell kubectl get pods --namespace=nginx-ingress ``` For more information, see the [Configuration guide](/nic/integrations/app-protect-waf-v5/configuration.md) and the NGINX Ingress Controller with App Protect version 5 example resources on GitHub [for VirtualServer resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/app-protect-waf-v5). ## Alternatives to building your own image {#pre-built-images} If you prefer not to build your own NGINX Ingress Controller image, you can use pre-built images. Here are your options: - Download the image using your NGINX Ingress Controller subscription certificate and key. View the [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) topic. - The [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) topic describes how to use your subscription JWT token to get the image. -------------------------------------------------------------------------------- title: "Build NGINX Ingress Controller with F5 WAF for NGINX" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-waf/installation/index.md -------------------------------------------------------------------------------- This document explains how to build a F5 NGINX Ingress Controller image with F5 WAF for NGINX from source code. **Note:** If you'd rather not build your own NGINX Ingress Controller image, see the [pre-built image options](#pre-built-images) at the end of this guide. ## Before you start - To use F5 WAF for NGINX with NGINX Ingress Controller, you must have NGINX Plus. - F5 WAF for NGINX package based installation (previously F5 WAF for NGINX v4) is not supported when `readOnlyRootFilesystem` is enabled. ## Prepare the environment Get your system ready for building and pushing the NGINX Ingress Controller image with F5 WAF for NGINX. 1. Sign in to your private registry. Replace `` with the path to your own private registry. ```shell docker login ``` 1. Clone the NGINX Ingress Controller repository: ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch v cd kubernetes-ingress ``` ## Build the image Follow these steps to build the NGINX Controller Image with F5 WAF for NGINX. 1. Place your NGINX Plus license files (_nginx-repo.crt_ and _nginx-repo.key_) in the project's root folder. To verify they're in place, run: ```shell ls nginx-repo.* ``` You should see: ```shell nginx-repo.crt nginx-repo.key ``` 2. Build the image. Replace `` with your chosen build option and `` with your private registry's path. Refer to the [Makefile targets](#makefile-targets) table below for the list of build options. ```shell make PREFIX=/nginx-plus-ingress TARGET=download ``` For example, to build a Debian-based image with NGINX Plus and F5 DoS for NGINX, run: ```shell make debian-image-dos-plus PREFIX=/nginx-plus-ingress TARGET=download ``` **What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](/nic/install/build.md#makefile-details). This version number is used for tracking and deployment purposes. **Note:** In the event a patch of NGINX Plus is released, make sure to rebuild your image to get the latest version. If your system is caching the Docker layers and not updating the packages, add `DOCKER_BUILD_OPTIONS="--pull --no-cache"` to the make command. ### Makefile targets {#makefile-targets} | Makefile Target | Description | Compatible Systems | |---------------------------|-------------------------------------------------------------------|---------------------| | **debian-image-nap-plus** | Builds a Debian-based image with NGINX Plus and the [F5 WAF for NGINX](/nginx-app-protect-waf/) module. | Debian | | **debian-image-nap-dos-plus** | Builds a Debian-based image with NGINX Plus, [F5 WAF for NGINX](/nginx-app-protect-waf/), and [F5 DoS for NGINX](/nginx-app-protect-dos/) | Debian | | **ubi-image-nap-plus** | Builds a UBI-based image with NGINX Plus and the [F5 WAF for NGINX](/nginx-app-protect-waf/) module. | OpenShift | | **ubi-image-nap-dos-plus** | Builds a UBNI-based image with NGINX Plus, [F5 WAF for NGINX](/nginx-app-protect-waf/), and [F5 DoS for NGINX](/nginx-app-protect-dos/). | OpenShift | **Note:** For the complete list of _Makefile_ targets and customizable variables, see the [Build NGINX Ingress Controller](/nic/install/build.md#makefile-details) topic. ## Push the image to your private registry Once you've successfully built the NGINX Ingress Controller image with F5 WAF for NGINX, the next step is to upload it to your private Docker registry. This makes the image available for deployment to your Kubernetes cluster. To upload the image, run the following command. If you're using a custom tag, add `TAG=your-tag` to the end of the command. Replace `` with your private registry's path. ```shell make push PREFIX=/nginx-plus-ingress ``` ## Set up role-based access control (RBAC) {#set-up-rbac} **Note:** To complete these steps you need admin access to your cluster. Refer to your Kubernetes platform's documentation to set up admin access. For Google Kubernetes Engine (GKE), you can refer to their [Role-Based Access Control guide](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control). 1. Create a namespace and a service account: ```shell kubectl apply -f deployments/common/ns-and-sa.yaml ``` 2. Create a cluster role and binding for the service account: ```shell kubectl apply -f deployments/rbac/rbac.yaml ```
If you're planning to use F5 WAF for NGINX or F5 DoS for NGINX, additional roles and bindings are needed. 1. (F5 WAF for NGINX only) Create the *App Protect* role and binding: ```shell kubectl apply -f deployments/rbac/ap-rbac.yaml ``` 2. (F5 DoS for NGINX only) Create the *App Protect DoS* role and binding: ```shell kubectl apply -f deployments/rbac/apdos-rbac.yaml ``` ## Create common resources {#create-common-resources} In this section, you'll create resources that most NGINX Ingress Controller installations require: 1. (Optional) Create a secret for the default NGINX server's TLS certificate and key. Complete this step only if you're using the [default server TLS secret](/nic/configuration/global-configuration/command-line-arguments#cmdoption-default-server-tls-secret.md) command-line argument. If you're not, feel free to skip this step. By default, the server returns a _404 Not Found_ page for all requests when no ingress rules are set up. Although we provide a self-signed certificate and key for testing purposes, we recommend using your own certificate. ```shell make secrets kubectl apply -f examples/shared-examples/default-server-secret/default-server-secret.yaml ``` 2. Create a ConfigMap to customize your NGINX settings: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` 3. Create an `IngressClass` resource. NGINX Ingress Controller won't start without an `IngressClass` resource. ```shell kubectl apply -f deployments/common/ingress-class.yaml ``` If you want to make this NGINX Ingress Controller instance your cluster's default, uncomment the `ingressclass.kubernetes.io/is-default-class` annotation. This action will auto-assign `IngressClass` to new ingresses that don't specify an `ingressClassName`. ## Create core custom resources {#create-custom-resources} To make sure your NGINX Ingress Controller pods reach the `Ready` state, you'll need to create custom resource definitions (CRDs) for various components. Alternatively, you can disable this requirement by setting the `-enable-custom-resources` command-line argument to `false`. There are two ways you can install the custom resource definitions: 1. Using a URL to apply a single CRD yaml file, which we recommend. 1. Applying your local copy of the CRD yaml files, which requires you to clone the repository. The core custom CRDs are the following: - [VirtualServer and VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) - [TransportServer](/nic/configuration/transportserver-resource.md) - [Policy](/nic/configuration/policy-resource/before-you-begin.md) - [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) #### Install CRDs from single YAML ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` #### Install CRDs after cloning the repo **Note:** Read the steps outlined in [Upgrade from 3.x to 4.x](/nic/install/upgrade.md#upgrade-from-3x-to-4x) before running the CRD upgrade and perform the steps if applicable. ```shell kubectl apply -f config/crd/bases/k8s.nginx.org_virtualservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_virtualserverroutes.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_transportservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_policies.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_globalconfigurations.yaml ``` ## Create F5 WAF for NGINX custom resources **Note:** If you're using NGINX Ingress Controller with the F5 WAF for NGINX module and policy bundles, you can skip this section. You will need to create and configure [Persistent Volume and Persistent Volume Claim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) in your Kubernetes cluster. #### Install CRDs from single YAML This single YAML file creates CRDs for the following resources: - `APPolicy` - `APLogConf` - `APUserSig` ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds-nap-waf.yaml ``` #### Install CRDs after cloning the repo **Note:** If you are installing the CRDs this way, ensure you have first cloned the repository. These YAML files create CRDs for the following resources: - `APPolicy` - `APLogConf` - `APUserSig` ```shell kubectl apply -f config/crd/bases/appprotect.f5.com_appolicies.yaml kubectl apply -f config/crd/bases/appprotect.f5.com_aplogconfs.yaml kubectl apply -f config/crd/bases/appprotect.f5.com_apusersigs.yaml ``` ## Deploy NGINX Ingress Controller {#deploy-ingress-controller} You have two options for deploying NGINX Ingress Controller: - **Deployment**. Choose this method for the flexibility to dynamically change the number of NGINX Ingress Controller replicas. - **DaemonSet**. Choose this method if you want NGINX Ingress Controller to run on all nodes or a subset of nodes. - **StatefulSet**. Choose this method when you need stable, persistent storage and ordered deployment/scaling for your NGINX Ingress Controller pods. Before you start, update the [command-line arguments](/nic/configuration/global-configuration/command-line-arguments.md) for the NGINX Ingress Controller container in the relevant manifest file to meet your specific requirements. **Note:** If you're using NGINX Ingress Controller with the AppProtect WAF module and policy bundles, you will need to modify the Deployment, DaemonSet, or StatefulSet file to include volumes and volume mounts. NGINX Ingress Controller **requires** the volume mount path to be `/etc/nginx/waf/bundles`. Add a `volumes` section to deployment template spec: ```yaml ... volumes: - name: persistentVolumeClaim: claimName: ... ``` Add volume mounts to the `containers` section: ```yaml ... volumeMounts: - name: mountPath: /etc/nginx/waf/bundles ... ``` **Note:** **StatefulSet Volume Configuration**: When using StatefulSet deployments, the `nginx-cache` volume is automatically provided via `volumeClaimTemplates` for persistent storage. F5 WAF for NGINX v5 volumes (like app-protect-config, app-protect-bundles) are still configured as regular volumes in the `volumes` section. Use `emptyDir` for temporary data or PersistentVolumeClaims if you need persistence for App Protect configurations across pod restarts. ### Using a Deployment For additional context on managing containers using Kubernetes Deployments, refer to the official Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) documentation. When you deploy NGINX Ingress Controller as a Deployment, Kubernetes automatically sets up a single NGINX Ingress Controller pod. - For NGINX, run: ```shell kubectl apply -f deployments/deployment/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/deployment/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a DaemonSet For additional context on managing containers using Kubernetes DaemonSets, refer to the official Kubernetes [DaemonSets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) documentation. When you deploy NGINX Ingress Controller as a DaemonSet, Kubernetes creates an Ingress Controller pod on every node in the cluster. - For NGINX, run: ```shell kubectl apply -f deployments/daemon-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/daemon-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a StatefulSet For additional context on managing containers using Kubernetes StatefulSets, refer to the official Kubernetes [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) documentation. When you deploy NGINX Ingress Controller as a StatefulSet, Kubernetes creates pods with stable network identities and persistent storage. - For NGINX, run: ```shell kubectl apply -f deployments/stateful-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/stateful-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. **Note:** StatefulSets include persistent volume claims for nginx cache storage via `volumeClaimTemplates`. You may need to configure a StorageClass in your cluster or modify the volumeClaimTemplates section in the manifest to match your storage requirements. Other volumes (like those needed for App Protect modules) are configured in the regular `volumes` section, not in volumeClaimTemplates. ## Enable F5 WAF for NGINX module (requires NGINX Plus) To enable the F5 WAF for NGINX Module: - Add the `--enable-app-protect` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-app-protect) to your Deployment, DaemonSet, or StatefulSet file. Additional WAF modules can be enabled by adding the appropriate command-line arguments: - To enable F5 WAF DoS Protection, add the `--enable-app-protect-dos` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-app-protect-dos). - To enable F5 WAF IP Intelligence, add the `--enable-app-protect-ip-intelligence` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-app-protect-ip-intelligence). ## Confirm NGINX Ingress Controller is running To confirm the NGINX Ingress Controller pods are operational, run: ```shell kubectl get pods --namespace=nginx-ingress ``` For more information, see the [Configuration guide](/nic/integrations/app-protect-waf/configuration.md) and the NGINX Ingress Controller with App Protect example resources on GitHub [for VirtualServer resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/app-protect-waf) and [for Ingress resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-waf). ## Alternatives to building your own image {#pre-built-images} If you prefer not to build your own NGINX Ingress Controller image, you can use pre-built images. Here are your options: - Download the image using your NGINX Ingress Controller subscription certificate and key. View the [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) topic. - The [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) topic describes how to use your subscription JWT token to get the image. -------------------------------------------------------------------------------- title: "Command-line arguments" url: https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/command-line-arguments/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller supports several command-line arguments, which are set based on installation method: - If you're using *Kubernetes Manifests* to install NGINX Ingress Controller, modify the Manifests to set the command-line arguments. View the [Installation with Manifests](/nic/install/manifests.md) topic for more information. - If you're using *Helm* to install NGINX Ingress Controller, modify the parameters of the Helm chart to set the command-line arguments. View the [Installation with Helm](/nic/install/helm.md) topic for more information. ### -enable-snippets Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources. Default `false`. ### -default-server-tls-secret `` Secret with a TLS certificate and key for TLS termination of the default server. - If not set, certificate and key in the file `/etc/nginx/secrets/default` are used. - If `/etc/nginx/secrets/default` doesn't exist, NGINX Ingress Controller will configure NGINX to reject TLS connections to the default server. - If a secret is set, but NGINX Ingress Controller is not able to fetch it from Kubernetes API, or it is not set and NGINX Ingress Controller fails to read the file "/etc/nginx/secrets/default", NGINX Ingress Controller will fail to start. Format: `/` When `-allow-empty-ingress-host` is enabled, this secret also provides TLS for Ingress resources without a `host`. ### -wildcard-tls-secret `` A Secret with a TLS certificate and key for TLS termination of every Ingress/VirtualServer host for which TLS termination is enabled but the Secret is not specified. - If the argument is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection - If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. Format: `/` ### -enable-custom-resources Enables custom resources. Default `true`. ### -enable-oidc Enables OIDC policies. Default `false`. ### -enable-leader-election Enables Leader election to avoid multiple replicas of the controller reporting the status of Ingress, VirtualServer and VirtualServerRoute resources -- only one replica will report status. Default `true`. See [-report-ingress-status](#cmdoption-report-ingress-status) flag. ### -enable-tls-passthrough Enable TLS Passthrough on port 443. Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -tls-passthrough-port `` Set the port for TLS Passthrough. Format: `[1024 - 65535]` (default `443`) Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -enable-cert-manager Enable x509 automated certificate management for VirtualServer resources using cert-manager (cert-manager.io). Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -enable-config-safety **Note:** This is an experimental feature. Behavior and configuration may change in future releases. When this feature is turned on, users can experience delayed pod startup times as resources are validated prior to being written. Delays are directly proportional to the number and complexity of resources. The `-enable-config-safety` flag enhances the stability and reliability of the NGINX Ingress Controller. When turned on, this feature ensures the validation of new configurations before reloading nginx. If a configuration is deemed invalid, the new config for the associated resource is rejected and the old working config is restored. By Default it is set to `false`. ### -enable-external-dns Enable integration with ExternalDNS for configuring public DNS entries for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -external-service `` Specifies the name of the service with the type LoadBalancer through which the NGINX Ingress Controller pods are exposed externally. The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). ### -ingresslink `` Specifies the name of the IngressLink resource, which exposes the NGINX Ingress Controller pods via a BIG-IP system. The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). ### -global-configuration `` A GlobalConfiguration resource for global configuration of NGINX Ingress Controller. Format: `/` Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -health-status Adds a location "/nginx-health" to the default server. The location responds with the 200 status code for any request. Useful for external health-checking of NGINX Ingress Controller. ### -health-status-uri `` Sets the URI of health status location in the default server. Requires [-health-status](#cmdoption-health-status). (default `/nginx-health`) ### -ingress-class `` The `-ingress-class` argument refers to the name of the resource `kind: IngressClass`. An IngressClass resource with a name equal to the class must be deployed. Otherwise, NGINX Ingress Controller will fail to start. NGINX Ingress Controller will only process Ingress resources that belong to its class (Whose `ingressClassName` value matches the value of `-ingress-class`), skipping the ones without it. It will also process all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the `ingressClassName` field. Default `nginx`. ### -ingress-template-path `` Path to the ingress NGINX configuration template for an ingress resource. Default for NGINX is `nginx.ingress.tmpl`; default for NGINX Plus is `nginx-plus.ingress.tmpl`. ### -allow-empty-ingress-host {#cmdoption-allow-empty-ingress-host} Allows Ingress resources to omit the `host` field. Default `false`. If multiple Ingress resources without a `host` conflict, NGINX Ingress Controller resolves the collision using the [winner selection algorithm](/nic/configuration/host-and-listener-collisions.md). To use multiple Ingress resources without a `host` across namespaces without conflict, use [mergeable Ingress](/nic/configuration/ingress-resources/cross-namespace-configuration.md). To configure TLS, use the command-line argument [`-default-server-tls-secret`](#cmdoption-default-server-tls-secret). To configure listener ports, use the command-line arguments [`-default-http-listener-port`](#cmdoption-default-http-listener-port) or [`-default-https-listener-port`](#cmdoption-default-https-listener-port). ### -leader-election-lock-name `` Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. Requires [-enable-leader-election](#cmdoption-enable-leader-election). ### -log_backtrace_at `` When logging hits line `file:N`, emit a stack trace. ### -main-template-path `` Path to the main NGINX configuration template. - Default for NGINX is `nginx.tmpl`. - Default for NGINX Plus is `nginx-plus.tmpl`. ### -nginx-configmaps `` A ConfigMap resource for customizing NGINX configuration. If a ConfigMap is set, but NGINX Ingress Controller is not able to fetch it from Kubernetes API, NGINX Ingress Controller will fail to start. Format: `/` ### -mgmt-configmap `` The Management ConfigMap resource is used for customizing the NGINX mgmt block. If using NGINX Plus, a Management ConfigMap must be set. If NGINX Ingress Controller is not able to fetch it from Kubernetes API, NGINX Ingress Controller will fail to start. Format: `/` ### -nginx-debug Enable debugging for NGINX. Uses the nginx-debug binary. Requires 'error-log-level: debug' in the ConfigMap. ### -nginx-plus Enable support for NGINX Plus. ### -nginx-reload-timeout `` Timeout in milliseconds which NGINX Ingress Controller will wait for a successful NGINX reload after a change or at the initial start. Default is 60000. ### -nginx-status Enable the NGINX stub_status, or the NGINX Plus API. Default `true`. ### -nginx-status-allow-cidrs `` Add IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas. (default `127.0.0.1,::1`) ### -nginx-status-port `` Set the port where the NGINX stub_status or the NGINX Plus API is exposed. Format: `[1024 - 65535]` (default `8080`) ### -proxy `` **Note:** This argument is intended for testing purposes only. Use a proxy server to connect to Kubernetes API started with `kubectl proxy`. NGINX Ingress Controller does not start NGINX and does not write any generated NGINX configuration files to disk. ### -report-ingress-status Updates the address field in the status of Ingress resources. Requires the [-external-service](#cmdoption-external-service) or [-ingresslink](#cmdoption-ingresslink) flag, or the `external-status-address` key in the ConfigMap. ### -transportserver-template-path `` Path to the TransportServer NGINX configuration template for a TransportServer resource. - Default for NGINX is `nginx.transportserver.tmpl`. - Default for NGINX Plus is `nginx-plus.transportserver.tmpl`. ### -log-level `` Log level for Ingress Controller logs. Allowed values: fatal, error, warn, info, debug, trace. - Default is `info`. ### -log-format `` Log format for Ingress Controller logs. Allowed values: glog, json, text. - Default is `glog`. ### -version Print the version, git-commit hash and build date and exit. ### -virtualserver-template-path `` Path to the VirtualServer NGINX configuration template for a VirtualServer resource. - Default for NGINX is `nginx.virtualserver.tmpl`. - Default for NGINX Plus is `nginx-plus.virtualserver.tmpl`. ### -vmodule `` A comma-separated list of pattern=N settings for file-filtered logging. ### -watch-namespace `` Comma separated list of namespaces NGINX Ingress Controller should watch for resources. By default NGINX Ingress Controller watches all namespaces. Mutually exclusive with "watch-namespace-label". ### -watch-namespace-label `` Configures NGINX Ingress Controller to watch only those namespaces with label foo=bar. By default NGINX Ingress Controller watches all namespaces. Mutually exclusive with "watch-namespace". ### -watch-secret-namespace `` Comma separated list of namespaces NGINX Ingress Controller should watch for secrets. If this arg is not configured, NGINX Ingress Controller watches the same namespaces for all resources, see "watch-namespace" and "watch-namespace-label". All namespaces included with this argument must be part of either `-watch-namespace` or `-watch-namespace-label`. ### -enable-prometheus-metrics Enables exposing NGINX or NGINX Plus metrics in the Prometheus format. ### -prometheus-metrics-listen-port `` Sets the port where the Prometheus metrics are exposed. Format: `[1024 - 65535]` (default `9113`) ### -prometheus-tls-secret `` A Secret with a TLS certificate and key for TLS termination of the Prometheus metrics endpoint. - If the argument is not set, the Prometheus endpoint will not use a TLS connection. - If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. ### -enable-service-insight Exposes the Service Insight endpoint for Ingress Controller. ### -service-insight-listen-port `` Sets the port where the Service Insight is exposed. Format: `[1024 - 65535]` (default `9114`) ### -service-insight-tls-secret `` A Secret with a TLS certificate and key for TLS termination of the Service Insight endpoint. - If the argument is not set, the Service Insight endpoint will not use a TLS connection. - If the argument is set, but NGINX Ingress Controller is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller will fail to start. Format: `/` ### -enable-latency-metrics Enable collection of latency metrics for upstreams. Requires [-enable-prometheus-metrics](#cmdoption-enable-prometheus-metrics). ### -enable-app-protect Enables support for App Protect. Requires [-nginx-plus](#cmdoption-nginx-plus). - If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will fail to start. ### -app-protect-log-level `` Sets log level for App Protect. Allowed values: fatal, error, warn, info, debug, trace. Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect](#cmdoption-enable-app-protect). - If the argument is set, but `nginx-plus` and `enable-app-protect` are set to false, NGINX Ingress Controller will fail to start. ### -enable-app-protect-dos Enables support for App Protect DoS. Requires [-nginx-plus](#cmdoption-nginx-plus). - If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will fail to start. ### -app-protect-dos-debug Enable debugging for App Protect DoS. Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). - If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. ### -app-protect-dos-max-daemons Max number of ADMD instances. Default `1`. Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). - If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. ### -app-protect-dos-max-workers Max number of nginx processes to support. Default `Number of CPU cores in the machine`. Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). - If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. ### -app-protect-dos-memory RAM memory size to consume in MB Default `50% of free RAM in the container or 80MB, the smaller`. Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect-dos](#cmdoption-enable-app-protect-dos). - If the argument is set, but `nginx-plus` and `enable-app-protect-dos` are set to false, NGINX Ingress Controller will fail to start. ### -enable-app-protect-ip-intelligence Enables App Protect IP Intelligence. Defaults to `false`. *Only applies to WAF single-container deployments*. Requires [-nginx-plus](#cmdoption-nginx-plus) and [-enable-app-protect](#cmdoption-enable-app-protect). ### -plm-storage-url SeaweedFS S3 endpoint from which NGINX Ingress Controller fetches the policy and logconf bundle. Leave empty to turn off PLM support. ### -plm-storage-credentials-secret A Secret containing the SeaweedFS admin secret in the `seaweedfs_admin_secret` key. Format: `/` ### -plm-storage-ca-secret An optional Secret containing `ca.crt` for SeaweedFS TLS verification. Format: `/` ### -plm-storage-client-ssl-secret An optional Secret containing `tls.crt` and `tls.key` for SeaweedFS mTLS. Format: `/` ### -plm-storage-insecure-skip-verify Turns off SeaweedFS TLS verification. For development and testing only. ### -ready-status Enables the readiness endpoint `/nginx-ready`. The endpoint returns a success code when NGINX has loaded all the config after the startup. Default `true`. ### -ready-status-port The HTTP port for the readiness endpoint. Format: `[1024 - 65535]` (default `8081`) ### -disable-ipv6 Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack. Default `false`. ### -default-http-listener-port Sets the port for the HTTP `default_server` listener. Default `80`. When [-allow-empty-ingress-host](#cmdoption-allow-empty-ingress-host) is enabled, also configures the HTTP listener port for Ingress resources without a `host`. ### -default-https-listener-port Sets the port for the HTTPS `default_server` listener. Default `443`. When [-allow-empty-ingress-host](#cmdoption-allow-empty-ingress-host) is enabled, also configures the HTTPS listener port for Ingress resources without a `host`. ### -ssl-dynamic-reload Used to activate or deactivate lazy loading for SSL Certificates. The default value is `true`. ### -weight-changes-dynamic-reload Enables the ability to change the weight distribution of two-way split clients without reloading NGINX. Requires [-nginx-plus](#cmdoption-nginx-plus). Using this feature may require increasing `map_hash_bucket_size`, `map_hash_max_size`, `variable_hash_bucket_size`, and `variable_hash_max_size` in the ConfigMap based on the number of two-way splits. The default value is `false`. - If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller will ignore the flag. ### -enable-directive-autoadjust Automatically adjusts NGINX buffer directives to prevent configuration errors. The default value is `false`. When enabled, the controller automatically adjusts `proxy_buffers`, `proxy_buffer_size`, and `proxy_busy_buffers_size` to ensure they work together properly and NGINX can start successfully. More explanation about this feature can be found in the guide [here](/nic/configuration/proxy-buffers-configuration.md). ### -enable-telemetry-reporting Enable gathering and reporting of software telemetry. The default value is `true`. ### -agent Enable NGINX Agent which can used with `-enable-app-protect` to send events to Security Monitoring. The default value is `false`. ### -agent-instance-group Specify the instance group name to use for the NGINX Ingress Controller deployment when using `-agent`. -------------------------------------------------------------------------------- title: "Command-line arguments" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/global-configuration/command-line-arguments/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller LTS supports several command-line arguments, which are set based on installation method: - If you're using *Kubernetes Manifests* to install NGINX Ingress Controller LTS, modify the Manifests to set the command-line arguments. View the [Installation with Manifests](/nic/lts/install/manifests.md) topic for more information. - If you're using *Helm* to install NGINX Ingress Controller LTS, modify the parameters of the Helm chart to set the command-line arguments. View the [Installation with Helm](/nic/lts/install/helm.md) topic for more information. ### -enable-snippets Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources. Default `false`. ### -default-server-tls-secret `` Secret with a TLS certificate and key for TLS termination of the default server. - If not set, certificate and key in the file `/etc/nginx/secrets/default` are used. - If `/etc/nginx/secrets/default` doesn't exist, NGINX Ingress Controller LTS will configure NGINX to reject TLS connections to the default server. - If a secret is set, but NGINX Ingress Controller LTS is not able to fetch it from Kubernetes API, or it is not set and NGINX Ingress Controller LTS fails to read the file "/etc/nginx/secrets/default", NGINX Ingress Controller LTS will fail to start. Format: `/` ### -wildcard-tls-secret `` A Secret with a TLS certificate and key for TLS termination of every Ingress/VirtualServer host for which TLS termination is enabled but the Secret is not specified. - If the argument is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection - If the argument is set, but NGINX Ingress Controller LTS is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller LTS will fail to start. Format: `/` ### -enable-custom-resources Enables custom resources. Default `true`. ### -enable-oidc Enables OIDC policies. Default `false`. ### -enable-leader-election Enables Leader election to avoid multiple replicas of the controller reporting the status of Ingress, VirtualServer and VirtualServerRoute resources -- only one replica will report status. Default `true`. See [-report-ingress-status](#cmdoption-report-ingress-status) flag. ### -enable-tls-passthrough Enable TLS Passthrough on port 443. Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -tls-passthrough-port `` Set the port for TLS Passthrough. Format: `[1024 - 65535]` (default `443`) Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -enable-cert-manager Enable x509 automated certificate management for VirtualServer resources using cert-manager (cert-manager.io). Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -enable-external-dns Enable integration with ExternalDNS for configuring public DNS entries for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -external-service `` Specifies the name of the service with the type LoadBalancer through which the NGINX Ingress Controller LTS pods are exposed externally. The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). ### -ingresslink `` Specifies the name of the IngressLink resource, which exposes the NGINX Ingress Controller LTS pods via a BIG-IP system. The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. For Ingress resources only: Requires [-report-ingress-status](#cmdoption-report-ingress-status). ### -global-configuration `` A GlobalConfiguration resource for global configuration of NGINX Ingress Controller LTS. Format: `/` Requires [-enable-custom-resources](#cmdoption-enable-custom-resources). ### -health-status Adds a location "/nginx-health" to the default server. The location responds with the 200 status code for any request. Useful for external health-checking of NGINX Ingress Controller LTS. ### -health-status-uri `` Sets the URI of health status location in the default server. Requires [-health-status](#cmdoption-health-status). (default `/nginx-health`) ### -ingress-class `` The `-ingress-class` argument refers to the name of the resource `kind: IngressClass`. An IngressClass resource with a name equal to the class must be deployed. Otherwise, NGINX Ingress Controller LTS will fail to start. NGINX Ingress Controller LTS will only process Ingress resources that belong to its class (Whose `ingressClassName` value matches the value of `-ingress-class`), skipping the ones without it. It will also process all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the `ingressClassName` field. Default `nginx`. ### -ingress-template-path `` Path to the ingress NGINX configuration template for an ingress resource. Default: `nginx-plus.ingress.tmpl`. ### -leader-election-lock-name `` Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. Requires [-enable-leader-election](#cmdoption-enable-leader-election). ### -log_backtrace_at `` When logging hits line `file:N`, emit a stack trace. ### -main-template-path `` Path to the main NGINX configuration template. - Default: `nginx-plus.tmpl`. ### -nginx-configmaps `` A ConfigMap resource for customizing NGINX configuration. If a ConfigMap is set, but NGINX Ingress Controller LTS is not able to fetch it from Kubernetes API, NGINX Ingress Controller LTS will fail to start. Format: `/` ### -mgmt-configmap `` The Management ConfigMap resource is used for customizing the NGINX mgmt block. If using NGINX Plus, a Management ConfigMap must be set. If NGINX Ingress Controller LTS is not able to fetch it from Kubernetes API, NGINX Ingress Controller LTS will fail to start. Format: `/` ### -nginx-debug Enable debugging for NGINX. Uses the nginx-debug binary. Requires 'error-log-level: debug' in the ConfigMap. ### -nginx-plus Enable support for NGINX Plus. ### -nginx-reload-timeout `` Timeout in milliseconds which NGINX Ingress Controller LTS will wait for a successful NGINX reload after a change or at the initial start. Default is 60000. ### -nginx-status Enable the NGINX Plus API. Default `true`. ### -nginx-status-allow-cidrs `` Add IP/CIDR blocks to the allow list for the NGINX Plus API. Separate multiple IP/CIDR by commas. (default `127.0.0.1,::1`) ### -nginx-status-port `` Set the port where the NGINX Plus API is exposed. Format: `[1024 - 65535]` (default `8080`) ### -proxy `` **warning:** This argument is intended for testing purposes only. Use a proxy server to connect to Kubernetes API started with `kubectl proxy`. NGINX Ingress Controller LTS does not start NGINX and does not write any generated NGINX configuration files to disk. ### -report-ingress-status Updates the address field in the status of Ingress resources. Requires the [-external-service](#cmdoption-external-service) or [-ingresslink](#cmdoption-ingresslink) flag, or the `external-status-address` key in the ConfigMap. ### -transportserver-template-path `` Path to the TransportServer NGINX configuration template for a TransportServer resource. - Default: `nginx-plus.transportserver.tmpl`. ### -log-level `` Log level for Ingress Controller logs. Allowed values: fatal, error, warn, info, debug, trace. - Default is `info`. ### -log-format `` Log format for Ingress Controller logs. Allowed values: glog, json, text. - Default is `glog`. ### -version Print the version, git-commit hash and build date and exit. ### -virtualserver-template-path `` Path to the VirtualServer NGINX configuration template for a VirtualServer resource. - Default: `nginx-plus.virtualserver.tmpl`. ### -vmodule `` A comma-separated list of pattern=N settings for file-filtered logging. ### -watch-namespace `` Comma separated list of namespaces NGINX Ingress Controller LTS should watch for resources. By default NGINX Ingress Controller LTS watches all namespaces. Mutually exclusive with "watch-namespace-label". ### -watch-namespace-label `` Configures NGINX Ingress Controller LTS to watch only those namespaces with label foo=bar. By default NGINX Ingress Controller LTS watches all namespaces. Mutually exclusive with "watch-namespace". ### -watch-secret-namespace `` Comma separated list of namespaces NGINX Ingress Controller LTS should watch for secrets. If this arg is not configured, NGINX Ingress Controller LTS watches the same namespaces for all resources, see "watch-namespace" and "watch-namespace-label". All namespaces included with this argument must be part of either `-watch-namespace` or `-watch-namespace-label`. ### -enable-prometheus-metrics Enables exposing NGINX Plus metrics in the Prometheus format. ### -prometheus-metrics-listen-port `` Sets the port where the Prometheus metrics are exposed. Format: `[1024 - 65535]` (default `9113`) ### -prometheus-tls-secret `` A Secret with a TLS certificate and key for TLS termination of the Prometheus metrics endpoint. - If the argument is not set, the Prometheus endpoint will not use a TLS connection. - If the argument is set, but NGINX Ingress Controller LTS is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller LTS will fail to start. ### -enable-service-insight Exposes the Service Insight endpoint for Ingress Controller. ### -service-insight-listen-port `` Sets the port where the Service Insight is exposed. Format: `[1024 - 65535]` (default `9114`) ### -service-insight-tls-secret `` A Secret with a TLS certificate and key for TLS termination of the Service Insight endpoint. - If the argument is not set, the Service Insight endpoint will not use a TLS connection. - If the argument is set, but NGINX Ingress Controller LTS is not able to fetch the Secret from Kubernetes API, NGINX Ingress Controller LTS will fail to start. Format: `/` ### -spire-agent-address `` Specifies the address of a running Spire agent. **For use with NGINX Service Mesh only**. - If the argument is set, but NGINX Ingress Controller LTS is unable to connect to the Spire Agent, NGINX Ingress Controller LTS will fail to start. ### -enable-internal-routes Enable support for internal routes with NGINX Service Mesh. **For use with NGINX Service Mesh only**. Requires [-spire-agent-address](#cmdoption-spire-agent-address). - If the argument is set, but `spire-agent-address` is not provided, NGINX Ingress Controller LTS will fail to start. ### -enable-latency-metrics Enable collection of latency metrics for upstreams. Requires [-enable-prometheus-metrics](#cmdoption-enable-prometheus-metrics). ### -ready-status Enables the readiness endpoint `/nginx-ready`. The endpoint returns a success code when NGINX has loaded all the config after the startup. Default `true`. ### -ready-status-port The HTTP port for the readiness endpoint. Format: `[1024 - 65535]` (default `8081`) ### -disable-ipv6 Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack. Default `false`. ### -default-http-listener-port Sets the port for the HTTP `default_server` listener. Default `80`. ### -default-https-listener-port Sets the port for the HTTPS `default_server` listener. Default `443`. ### -ssl-dynamic-reload Used to activate or deactivate lazy loading for SSL Certificates. The default value is `true`. ### -weight-changes-dynamic-reload Enables the ability to change the weight distribution of two-way split clients without reloading NGINX. Requires [-nginx-plus](#cmdoption-nginx-plus). Using this feature may require increasing `map_hash_bucket_size`, `map_hash_max_size`, `variable_hash_bucket_size`, and `variable_hash_max_size` in the ConfigMap based on the number of two-way splits. The default value is `false`. - If the argument is set, but `nginx-plus` is set to false, NGINX Ingress Controller LTS will ignore the flag. ### -enable-directive-autoadjust Automatically adjusts NGINX buffer directives to prevent configuration errors. The default value is `false`. When enabled, the controller automatically adjusts `proxy_buffers`, `proxy_buffer_size`, and `proxy_busy_buffers_size` to ensure they work together properly and NGINX can start successfully. More explanation about this feature can be found in the guide [here](/nic/lts/configuration/proxy-buffers-configuration.md). ### -enable-telemetry-reporting Enable gathering and reporting of software telemetry. The default value is `true`. -------------------------------------------------------------------------------- title: "Commercial support" url: https://docs.nginx.com/nginx-ingress-controller/lts/troubleshooting/troubleshoot-support/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller LTS adheres to the support policy detailed in the following knowledge base article: [K000140156](https://my.f5.com/manage/s/article/K000140156). After opening a support ticket, F5 staff will request additional information to better understand the problem. The [nginx-supportpkg-for-k8s](https://github.com/nginxinc/nginx-supportpkg-for-k8s) plugin collects the information needed by F5 Technical Support to assist with troubleshooting your issue. When used, the plugin will generate a tarball of the collected information which can be shared with the support channels. The plugin uses [krew](https://krew.sigs.k8s.io), the plugin manager for the Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/) command-line tool. The plugin may collect some or all of the following global and namespace-specific information: * K8s version, nodes information, and Custom Resources (kubectl describe output) * Pods' logs * List of Pods, events, ConfigMaps, Services, Deployments, Daemonsets, StatefulSets, ReplicaSets, and Leases * K8s metrics * Helm deployments * `nginx -T` output from NGINX-related pods This plugin **does not** collect secrets or coredumps. Visit the [project’s GitHub repository](https://github.com/nginxinc/nginx-supportpkg-for-k8s) for further details. ## Support channels - If you experience issues with NGINX Ingress Controller LTS, please [open an issue](https://github.com/nginx/kubernetes-ingress/issues/new?assignees=&labels=bug%2Cneeds+triage&projects=&template=BUG-REPORT.yml&title=%5BBug%5D%3A+) in GitHub. - If you have any enhancement requests, please [open a feature request](https://github.com/nginx/kubernetes-ingress/issues/new?assignees=&labels=proposal&projects=&template=feature_request.md&title=) in GitHub. - If you have any ideas or suggestions to discuss, please [open an idea discussion](https://github.com/nginx/kubernetes-ingress/discussions/categories/ideas) in GitHub. -------------------------------------------------------------------------------- title: "Commercial support" url: https://docs.nginx.com/nginx-ingress-controller/troubleshooting/troubleshoot-support/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller adheres to the support policy detailed in the following knowledge base article: [K000140156](https://my.f5.com/manage/s/article/K000140156). After opening a support ticket, F5 staff will request additional information to better understand the problem. The [nginx-supportpkg-for-k8s](https://github.com/nginxinc/nginx-supportpkg-for-k8s) plugin collects the information needed by F5 Technical Support to assist with troubleshooting your issue. When used, the plugin will generate a tarball of the collected information which can be shared with the support channels. The plugin uses [krew](https://krew.sigs.k8s.io), the plugin manager for the Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/) command-line tool. The plugin may collect some or all of the following global and namespace-specific information: * K8s version, nodes information, and Custom Resources (kubectl describe output) * Pods' logs * List of Pods, events, ConfigMaps, Services, Deployments, Daemonsets, StatefulSets, ReplicaSets, and Leases * K8s metrics * Helm deployments * `nginx -T` output from NGINX-related pods This plugin **does not** collect secrets or coredumps. Visit the [project’s GitHub repository](https://github.com/nginxinc/nginx-supportpkg-for-k8s) for further details. ## Support channels - If you experience issues with NGINX Ingress Controller, please [open an issue](https://github.com/nginx/kubernetes-ingress/issues/new?assignees=&labels=bug%2Cneeds+triage&projects=&template=BUG-REPORT.yml&title=%5BBug%5D%3A+) in GitHub. - If you have any enhancement requests, please [open a feature request](https://github.com/nginx/kubernetes-ingress/issues/new?assignees=&labels=proposal&projects=&template=feature_request.md&title=) in GitHub. - If you have any ideas or suggestions to discuss, please [open an idea discussion](https://github.com/nginx/kubernetes-ingress/discussions/categories/ideas) in GitHub. -------------------------------------------------------------------------------- title: "Download NGINX Ingress Controller from the F5 Registry" url: https://docs.nginx.com/nginx-ingress-controller/install/images/registry-download/index.md -------------------------------------------------------------------------------- This page describes how to download an F5 NGINX Plus Ingress Controller image from the official F5 Docker registry. The F5 Registry images include versions with F5 WAF for NGINX and F5 DoS for NGINX. ## Before you begin To follow these steps, you will need the following pre-requisites: - [Docker v18.09 or higher](https://docs.docker.com/engine/release-notes/18.09/) You can also get the NGINX Ingress Controller image using the following alternate methods: - [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) - [Build NGINX Ingress Controller](/nic/install/build.md) - For NGINX Open Source, you can pull [an image from DockerHub](https://hub.docker.com/r/nginx/nginx-ingress/) ### Download your subscription credential files In order to obtain a container image, you will need the JSON Web Token file or SSL certificate and private key files provided with your NGINX Plus subscription. These files grant access to the package repository from which the script will download the NGINX Plus package: #### JSON Web Token #### SSL ### Set up Docker for the F5 Container Registry This step describes how to use Docker to communicate with the F5 Container Registry located at `private-registry.nginx.com`. **Note:** The steps provided are for Linux. For Mac or Windows, see the [Docker for Mac](https://docs.docker.com/docker-for-mac/#add-client-certificates) or [Docker for Windows](https://docs.docker.com/docker-for-windows/#how-do-i-add-client-certificates) documentation. For more details on Docker Engine security, you can refer to the [Docker Engine Security documentation](https://docs.docker.com/engine/security/). #### JSON Web Token Open the JSON Web Token file previously downloaded from [MyF5](https://my.f5.com) customer portal (for example, `nginx-repo-12345abc.jwt`) and copy its contents. Log in to the Docker registry using the contents of the JSON Web Token file: ```shell docker login private-registry.nginx.com --username= --password=none ``` #### SSL Create a directory and copy your certificate and key to this directory: ```shell mkdir -p /etc/docker/certs.d/private-registry.nginx.com cp /etc/docker/certs.d/private-registry.nginx.com/client.cert cp /etc/docker/certs.d/private-registry.nginx.com/client.key ``` ## Pull the image Identify which image you need using the [Technical specifications](/nic/technical-specifications.md#images-with-nginx-plus) topic. Next, pull the image from `private-registry.nginx.com`. Replace `` with the specific version you need, for example, ``. - For NGINX Plus Ingress Controller, run: ```shell docker pull private-registry.nginx.com/nginx-ic/nginx-plus-ingress: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX, run: ```shell docker pull private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX and NGINX Agent 3 (required for NGINX One Console, available starting with NGINX Ingress Controller 5.5.0), run: ```shell docker pull private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:-agent ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX v5, run: ```shell docker pull private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress: ``` ```shell docker pull private-registry.nginx.com/nap/waf-config-mgr: ``` ```shell docker pull private-registry.nginx.com/nap/waf-enforcer: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX v5 and NGINX Agent 3 (required for NGINX One Console, available starting with NGINX Ingress Controller 5.5.0), run: ```shell docker pull private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress:-agent ``` ```shell docker pull private-registry.nginx.com/nap/waf-config-mgr: ``` ```shell docker pull private-registry.nginx.com/nap/waf-enforcer: ``` - For NGINX Plus Ingress Controller with F5 DoS for NGINX, run: ```shell docker pull private-registry.nginx.com/nginx-ic-dos/nginx-plus-ingress: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX and F5 DoS for NGINX, run: ```shell docker pull private-registry.nginx.com/nginx-ic-nap-dos/nginx-plus-ingress: ``` You can use the Docker registry API to list the available image tags by running the following commands. Replace `` with the location of your client key and `` with the location of your client certificate. The `jq` command was used in these examples to make the JSON output easier to read. ```shell curl https://private-registry.nginx.com/v2/nginx-ic/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic/nginx-plus-ingress", "tags": [ "-alpine", "-alpine-fips", "-ubi", "" ] } ``` ```shell curl https://private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic-nap/nginx-plus-ingress", "tags": [ "-alpine-fips", "-alpine-fips-agent", "-ubi", "-ubi-agent", "-ubi8", "-ubi8-agent", "", "-agent" ] } ``` ```shell curl https://private-registry.nginx.com/v2/nginx-ic-nap-v5/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic-nap-v5/nginx-plus-ingress", "tags": [ "-alpine-fips", "-alpine-fips-agent", "-ubi", "-ubi-agent", "-ubi8", "-ubi8-agent", "", "-agent" ] } ``` ```shell curl https://private-registry.nginx.com/v2/nginx-ic-dos/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic-dos/nginx-plus-ingress", "tags": [ "-ubi", "" ] } ``` ## Push to your private registry After pulling the image, tag it and upload it to your private registry. 1. Log in to your private registry: ```shell docker login ``` 1. Tag and push the image. Replace `` with your registry's path and `` with the version you're using, for example ``: - For NGINX Plus Ingress Controller, run: ```shell docker tag private-registry.nginx.com/nginx-ic/nginx-plus-ingress: /nginx-ic/nginx-plus-ingress: docker push /nginx-ic/nginx-plus-ingress: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX, run: ```shell docker tag private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress: /nginx-ic-nap/nginx-plus-ingress: docker push /nginx-ic-nap/nginx-plus-ingress: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX and NGINX Agent 3, run: ```shell docker tag private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress:-agent /nginx-ic-nap/nginx-plus-ingress:-agent docker push /nginx-ic-nap/nginx-plus-ingress:-agent ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX v5, run: ```shell docker tag private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress: /nginx-ic-nap-v5/nginx-plus-ingress: docker push /nginx-ic-nap-v5/nginx-plus-ingress: ``` ```shell docker tag private-registry.nginx.com/nap/waf-config-mgr: /nap/waf-config-mgr: docker push /nap/waf-config-mgr: ``` ```shell docker tag private-registry.nginx.com/nap/waf-enforcer: /nap/waf-enforcer: docker push /nap/waf-enforcer: ``` - For NGINX Plus Ingress Controller with F5 WAF for NGINX v5 and NGINX Agent 3, run: ```shell docker tag private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress:-agent /nginx-ic-nap-v5/nginx-plus-ingress:-agent docker push /nginx-ic-nap-v5/nginx-plus-ingress:-agent ``` ```shell docker tag private-registry.nginx.com/nap/waf-config-mgr: /nap/waf-config-mgr: docker push /nap/waf-config-mgr: ``` ```shell docker tag private-registry.nginx.com/nap/waf-enforcer: /nap/waf-enforcer: docker push /nap/waf-enforcer: ``` - For NGINX Plus Ingress Controller with F5 DoS for NGINX, run: ```shell docker tag private-registry.nginx.com/nginx-ic-dos/nginx-plus-ingress: /nginx-ic-dos/nginx-plus-ingress: docker push /nginx-ic-dos/nginx-plus-ingress: ``` ## Troubleshooting If you encounter issues while following this guide, here are some possible solutions: - **Certificate errors** - **Likely Cause**: Incorrect certificate or key location, or using an NGINX Plus certificate. - **Solution**: Verify you have the correct NGINX Ingress Controller certificate and key. Place them in the correct directory and ensure the certificate has a *.cert* extension. - **Docker version compatibility** - **Likely Cause**: Outdated Docker version. - **Solution**: Make sure you're running [Docker v18.09 or higher](https://docs.docker.com/engine/release-notes/18.09/). Upgrade if necessary. - **Can't pull the image** - **Likely Cause**: Mismatched image name or tag. - **Solution**: Double-check the image name and tag matches the [Technical specifications](/nic/technical-specifications.md#images-with-nginx-plus) document. - **Failed to push to private registry** - **Likely Cause**: Not logged into your private registry or incorrect image tagging. - **Solution**: Verify login status and correct image tagging before pushing. Consult the [Docker documentation](https://docs.docker.com/docker-hub/repos/) for more details. -------------------------------------------------------------------------------- title: "Download NGINX Ingress Controller LTS from the F5 Registry" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/images/registry-download/index.md -------------------------------------------------------------------------------- This page describes how to download an F5 NGINX Plus Ingress Controller LTS image from the official F5 Docker registry. ## Before you begin To follow these steps, you will need the following pre-requisites: - [Docker v18.09 or higher](https://docs.docker.com/engine/release-notes/18.09/) You can also get the NGINX Ingress Controller LTS image using the following alternate methods: - [Add an NGINX Ingress Controller LTS image to your cluster](/nic/lts/install/images/add-image-to-cluster.md) ### Download your subscription credential files In order to obtain a container image, you will need the JSON Web Token file or SSL certificate and private key files provided with your NGINX Plus subscription. These files grant access to the package repository from which the script will download the NGINX Plus package: #### JSON Web Token #### SSL ### Set up Docker for the F5 Container Registry This step describes how to use Docker to communicate with the F5 Container Registry located at `private-registry.nginx.com`. **Note:** The steps provided are for Linux. For Mac or Windows, see the [Docker for Mac](https://docs.docker.com/docker-for-mac/#add-client-certificates) or [Docker for Windows](https://docs.docker.com/docker-for-windows/#how-do-i-add-client-certificates) documentation. For more details on Docker Engine security, you can refer to the [Docker Engine Security documentation](https://docs.docker.com/engine/security/). #### JSON Web Token Open the JSON Web Token file previously downloaded from [MyF5](https://my.f5.com) customer portal (for example, `nginx-repo-12345abc.jwt`) and copy its contents. Log in to the Docker registry using the contents of the JSON Web Token file: ```shell docker login private-registry.nginx.com --username= --password=none ``` #### SSL Create a directory and copy your certificate and key to this directory: ```shell mkdir -p /etc/docker/certs.d/private-registry.nginx.com cp /etc/docker/certs.d/private-registry.nginx.com/client.cert cp /etc/docker/certs.d/private-registry.nginx.com/client.key ``` ## Pull the image Identify which image you need using the [Technical specifications](/nic/lts/technical-specifications.md#images-with-nginx-plus) topic. Next, pull the image from `private-registry.nginx.com`. Replace `` with the specific version you need, for example, ``. - For NGINX Plus Ingress Controller, run: ```shell docker pull private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress: ``` You can use the Docker registry API to list the available image tags by running the following commands. Replace `` with the location of your client key and `` with the location of your client certificate. The `jq` command was used in these examples to make the JSON output easier to read. ```shell curl https://private-registry.nginx.com/v2/nginx-ic/lts/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic/lts/nginx-plus-ingress", "tags": [ "" ] } ``` ## Push to your private registry After pulling the image, tag it and upload it to your private registry. 1. Log in to your private registry: ```shell docker login ``` 1. Tag and push the image. Replace `` with your registry's path and `` with the version you're using, for example ``: - For NGINX Ingress Controller LTS, run: ```shell docker tag private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress: /nginx-ic/lts/nginx-plus-ingress: docker push /nginx-ic/lts/nginx-plus-ingress: ``` ## Troubleshooting If you encounter issues while following this guide, here are some possible solutions: - **Certificate errors** - **Likely Cause**: Incorrect certificate or key location, or using an NGINX Plus certificate. - **Solution**: Verify you have the correct NGINX Ingress Controller LTS certificate and key. Place them in the correct directory and ensure the certificate has a *.cert* extension. - **Docker version compatibility** - **Likely Cause**: Outdated Docker version. - **Solution**: Make sure you're running [Docker v18.09 or higher](https://docs.docker.com/engine/release-notes/18.09/). Upgrade if necessary. - **Can't pull the image** - **Likely Cause**: Mismatched image name or tag. - **Solution**: Double-check the image name and tag matches the [Technical specifications](/nic/lts/technical-specifications.md#images-with-nginx-plus) document. - **Failed to push to private registry** - **Likely Cause**: Not logged into your private registry or incorrect image tagging. - **Solution**: Verify login status and correct image tagging before pushing. Consult the [Docker documentation](https://docs.docker.com/docker-hub/repos/) for more details. -------------------------------------------------------------------------------- title: "Logs available from NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/logging/index.md -------------------------------------------------------------------------------- This document gives an overview of logging provided by F5 NGINX Ingress Controller. NGINX Ingress Controller exposes the logs of the Ingress Controller process (The process that generates NGINX configuration and reloads NGINX to apply it) and NGINX access and error logs. All logs are sent to the standard output and error of the NGINX Ingress Controller process. To view the logs, you can execute the `kubectl logs` command for an Ingress Controller pod. For example: ```shell kubectl logs -n nginx-ingress ``` ## NGINX Ingress Controller Process Logs The NGINX Ingress Controller process logs are configured through the `-log-level` command-line argument of the NGINX Ingress Controller, which sets the log level. The default value is `info`. Other options include: `trace`, `debug`, `info`, `warning`, `error` and `fatal`. The value `debug` is useful for troubleshooting: you will be able to see how NGINX Ingress Controller gets updates from the Kubernetes API, generates NGINX configuration and reloads NGINX. Read more about NGINX Ingress Controller [command-line arguments](/nic/configuration/global-configuration/command-line-arguments.md). ## NGINX Logs NGINX includes two logs: - *Access log*, where NGINX writes information about client requests in the access log right after the request is processed. The access log is configured via the [logging-related](/nic/configuration/global-configuration/configmap-resource.md#logging) ConfigMap keys: - `log-format` for HTTP and HTTPS traffic. - `stream-log-format` for TCP, UDP, and TLS Passthrough traffic. Additionally, you can disable access logging with the `access-log-off` ConfigMap key. - *Error log*, where NGINX writes information about encountered issues of different severity levels. It is configured via the `error-log-level` [ConfigMap key](/nic/configuration/global-configuration.md#configmap-resource#logging). To enable debug logging, set the level to `debug` and also set the `-nginx-debug` [command-line argument](/nic/configuration/global-configuration.md#command-line-arguments), so that NGINX is started with the debug binary `nginx-debug`. Read more about [NGINX logs](/nginx/admin-guide/monitoring/logging.md) from NGINX Admin guide. -------------------------------------------------------------------------------- title: "Logs available from NGINX Ingress Controller LTS" url: https://docs.nginx.com/nginx-ingress-controller/lts/logging-and-monitoring/logging/index.md -------------------------------------------------------------------------------- This document gives an overview of logging provided by F5 NGINX Ingress Controller LTS. NGINX Ingress Controller LTS exposes the logs of the Ingress Controller process (The process that generates NGINX configuration and reloads NGINX to apply it) and NGINX access and error logs. All logs are sent to the standard output and error of the NGINX Ingress Controller LTS process. To view the logs, you can execute the `kubectl logs` command for an Ingress Controller pod. For example: ```shell kubectl logs -n nginx-ingress ``` ## NGINX Ingress Controller LTS Process Logs The NGINX Ingress Controller LTS process logs are configured through the `-log-level` command-line argument of the NGINX Ingress Controller LTS, which sets the log level. The default value is `info`. Other options include: `trace`, `debug`, `info`, `warning`, `error` and `fatal`. The value `debug` is useful for troubleshooting: you will be able to see how NGINX Ingress Controller LTS gets updates from the Kubernetes API, generates NGINX configuration and reloads NGINX. Read more about NGINX Ingress Controller LTS [command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments.md). ## NGINX Logs NGINX includes two logs: - *Access log*, where NGINX writes information about client requests in the access log right after the request is processed. The access log is configured via the [logging-related](/nic/lts/configuration/global-configuration/configmap-resource.md#logging) ConfigMap keys: - `log-format` for HTTP and HTTPS traffic. - `stream-log-format` for TCP, UDP, and TLS Passthrough traffic. Additionally, you can disable access logging with the `access-log-off` ConfigMap key. - *Error log*, where NGINX writes information about encountered issues of different severity levels. It is configured via the `error-log-level` [ConfigMap key](/nic/lts/configuration/global-configuration.md#configmap-resource#logging). To enable debug logging, set the level to `debug` and also set the `-nginx-debug` [command-line argument](/nic/lts/configuration/global-configuration.md#command-line-arguments), so that NGINX is started with the debug binary `nginx-debug`. Read more about [NGINX logs](/nginx/admin-guide/monitoring/logging.md) from NGINX Admin guide. -------------------------------------------------------------------------------- title: "Use Helm to Install NGINX Ingress Controller with NGINX Open Source" url: https://docs.nginx.com/nginx-ingress-controller/install/helm/open-source/index.md -------------------------------------------------------------------------------- This page describes how to use Helm to install F5 NGINX Ingress Controller with NGINX Open Source. It explains the requirements for NGINX Ingress Controller, how to obtain and install the Helm chart, and what custom resource definitions (CRDs) are installed during the process. By following these instructions, you will finish with a functional NGINX Ingress Controller instance for your Kubernetes cluster. ## Before you begin - A [supported Kubernetes version](/nic/technical-specifications.md#supported-kubernetes-versions) - A functional Kubernetes cluster - [Helm 3.19+.](https://helm.sh/docs/intro/install) Throughout this page, you will see placeholder values indicated with angular brackets, such as **\**. Replace them accordingly for your installation. **Note:** The `edge` version **is not intended for production use**. It is intended for testing and development purposes only. If you'd like to test the latest changes in NGINX Ingress Controller before a new release, you can install the `edge` version, which is built from the `main` branch of the [NGINX Ingress Controller repository](https://github.com/nginx/kubernetes-ingress). You can install the `edge` version by specifying the `--version` flag with the value `0.0.0-edge`: ```shell helm install oci://ghcr.io/nginx/charts/nginx-ingress --version 0.0.0-edge ``` ## Install the Helm chart You have two options for installing the Helm chart: directly from the OCI registry, or using the source. ### OCI Registry To install NGINX Ingress Controller using the OCI registry, run this command with your release name: ```shell helm install oci://ghcr.io/nginx/charts/nginx-ingress --version ``` #### Example output ```text Pulled: ghcr.io/nginx/charts/nginx-ingress: Digest: sha256:bb452d593c31b6be39f459f9604882e170227429821bac01e7ddd7da16d91ba1 NAME: h4-oss LAST DEPLOYED: Fri Nov 28 11:53:57 2025 NAMESPACE: default STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None NOTES: NGINX Ingress Controller has been installed. For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/ Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ ``` ### From source To install NGINX Ingress Controller from source, first pull the chart by running this command: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress --untar --version ``` #### Example output ```text Pulled: ghcr.io/nginx/charts/nginx-ingress: Digest: sha256:bb452d593c31b6be39f459f9604882e170227429821bac01e7ddd7da16d91ba1 ``` Then use the `cd` command to change your working directory to _nginx-ingress_: ```shell cd nginx-ingress ``` Finally, install the chart with your release name with `helm install`: ```shell helm install . ``` #### Example output ```text NAME: h4-oss-source LAST DEPLOYED: Fri Nov 28 12:06:07 2025 NAMESPACE: default STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None NOTES: NGINX Ingress Controller has been installed. For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/ Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ ``` ## Verify the deployment To verify that NGINX Ingress Controller has been installed correctly, you can review `ingressclasses` with `kubectl get`: ```shell kubectl get ingressclasses ``` #### Example output ```text NAME CONTROLLER PARAMETERS AGE nginx nginx.org/ingress-controller 10m ``` ## Custom Resource Definitions When installing the chart, Helm will install the required CRDs. Without them, NGINX Ingress Controller pods will not become _Ready_. If you do not use the custom resources that require those CRDs, add the parameter `--skip-crds` in your `helm install` command. The following chart parameters should be set to `false`: - `controller.enableCustomResources` - `controller.appprotect.enable` - `controller.appprotectdos.enable` ## Next steps - [NGINX Ingress Controller Helm chart parameters](/nic/install/helm/parameters.md) - [Security recommendations](/nic/configuration/security.md) - [Basic configuration](/nic/configuration/ingress-resources/basic-configuration.md) - [Extensibility with NGINX Plus](/nic/overview/nginx-plus.md) -------------------------------------------------------------------------------- title: "Add an NGINX Ingress Controller image to your cluster" url: https://docs.nginx.com/nginx-ingress-controller/install/images/add-image-to-cluster/index.md -------------------------------------------------------------------------------- This document describes how to add an F5 NGINX Plus Ingress Controller image from the F5 Docker registry into your Kubernetes cluster using a JWT token. ## Before you begin To follow these steps, you will need the following pre-requisite: - [Create a license Secret](/nic/install/license-secret.md) You can also get the NGINX Ingress Controller image using the following alternate methods: - [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) - [Build NGINX Ingress Controller](/nic/install/build.md) - For NGINX Open Source, you can pull the [nginx/nginx-ingress image](https://hub.docker.com/r/nginx/nginx-ingress/) from DockerHub ## Helm deployments If you are using Helm for deployment, there are two main methods: using a _chart_ or _source_. ### Add the image from chart The following command installs NGINX Ingress Controller with a Helm chart, passing required arguments using the `set` parameter. ```shell helm install my-release -n nginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress --version --set controller.image.repository=private-registry.nginx.com/nginx-ic/nginx-plus-ingress --set controller.image.tag= --set controller.nginxplus=true --set controller.serviceAccount.imagePullSecretName=regcred ``` You can also use the certificate and key from the MyF5 portal and the Docker registry API to list the available image tags for the repositories, for example: ```shell curl https://private-registry.nginx.com/v2/nginx-ic/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic/nginx-plus-ingress", "tags": [ "-alpine", "-alpine-fips", "-ubi", "" ] } ``` ```shell curl https://private-registry.nginx.com/v2/nginx-ic-nap/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic-nap/nginx-plus-ingress", "tags": [ "-alpine-fips", "-ubi", "" ] } ``` ```shell curl https://private-registry.nginx.com/v2/nginx-ic-dos/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic-dos/nginx-plus-ingress", "tags": [ "-ubi", "" ] } ``` The `jq` command was used in these examples to make the JSON output easier to read. ### Add the image from source The [Installation with Helm](/nic/install/helm.md#install-the-helm-chart-from-source) documentation has a section describing how to use sources: these are the unique steps for Docker secrets using JWT tokens. 1. Clone the NGINX [`kubernetes-ingress` repository](https://github.com/nginx/kubernetes-ingress). 1. Navigate to the `charts/nginx-ingress` folder of your local clone. 1. Open the `values.yaml` file in an editor. You must change a few lines NGINX Ingress Controller with NGINX Plus to be deployed. 1. Change the `nginxplus` argument to `true`. 1. Change the `repository` argument to the NGINX Ingress Controller image you intend to use. 1. Add an argument to `imagePullSecretName` or `imagePullSecretsNames` to allow Docker to pull the image from the private registry. The following code block shows snippets of the parameters you will need to change, and an example of their contents: ```yaml ## Deploys the Ingress Controller for NGINX Plus nginxplus: true ## Truncated fields ## ... ## ... image: ## The image repository for the desired NGINX Ingress Controller image repository: private-registry.nginx.com/nginx-ic/nginx-plus-ingress ## The version tag tag: serviceAccount: ## The annotations of the service account of the Ingress Controller pods. annotations: {} ## Truncated fields ## ... ## ... ## The name of the secret containing docker registry credentials. ## Secret must exist in the same namespace as the helm release. ## Note that also imagePullSecretsNames can be used here if multiple secrets need to be set. imagePullSecretName: regcred ``` With the modified `values.yaml` file, you can now use Helm to install NGINX Ingress Controller, for example: ```shell helm install nicdev01 -n nginx-ingress --create-namespace -f values.yaml . ``` The above command will install NGINX Ingress Controller in the `nginx-ingress` namespace. If the namespace does not exist, `--create-namespace` will create it. Using `-f values.yaml` tells Helm to use the `values.yaml` file that you modified earlier with the settings you want to apply for your NGINX Ingress Controller deployment. ## Manifest deployment The page ["Installation with Manifests"](/nic/install/manifests.md) explains how to install NGINX Ingress Controller using manifests. The following snippet is an example of a deployment: ```yaml spec: serviceAccountName: nginx-ingress imagePullSecrets: - name: regcred automountServiceAccountToken: true securityContext: seccompProfile: type: RuntimeDefault containers: - image: private-registry.nginx.com/nginx-ic/nginx-plus-ingress: imagePullPolicy: IfNotPresent name: nginx-plus-ingress ``` The `imagePullSecrets` and `containers.image` lines represent the Kubernetes secret, as well as the registry and version of NGINX Ingress Controller we are going to deploy. ## Download an image for local use If you need to download an image for local use (Such as to push to a different container registry), use this command: ```shell docker login private-registry.nginx.com --username= --password=none ``` Replace the contents of `` with the contents of the JWT token itself. Once you have successfully pulled the image, you can then tag it as needed. **Note:** For security, follow these practices with JSON Web Tokens (JWTs), passwords, and shell history: 1. **JWTs:** JWTs are sensitive information. Store them securely. Delete them after use to prevent unauthorized access. 1. **Shell history:** Commands that include JWTs or passwords are recorded in the history of your shell, in plain text. Clear your shell history after running such commands. For example, if you use bash, you can delete commands in your `~/.bash_history` file. Alternatively, you can run the `history -c` command to erase your shell history. Follow these practices to help ensure the security of your system and data. -------------------------------------------------------------------------------- title: "Add an NGINX Ingress Controller LTS image to your cluster" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/images/add-image-to-cluster/index.md -------------------------------------------------------------------------------- This document describes how to add an F5 NGINX Plus Ingress Controller image from the F5 Docker registry into your Kubernetes cluster using a JWT token. ## Before you begin To follow these steps, you will need the following pre-requisite: - [Create a license Secret](/nic/lts/install/license-secret.md) You can also get the NGINX Ingress Controller LTS image using the following alternate methods: - [Download NGINX Ingress Controller LTS from the F5 Registry](/nic/lts/install/images/registry-download.md) ## Helm deployments If you are using Helm for deployment, there are two main methods: using a _chart_ or _source_. ### Add the image from chart The following command installs NGINX Ingress Controller LTS with a Helm chart, passing required arguments using the `set` parameter. ```shell helm install my-release -n nginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress-lts --version --set controller.image.repository=private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress --set controller.image.tag= --set controller.nginxplus=true --set controller.serviceAccount.imagePullSecretName=regcred ``` You can also use the certificate and key from the MyF5 portal and the Docker registry API to list the available image tags for the repositories, for example: ```shell curl https://private-registry.nginx.com/v2/nginx-ic/lts/nginx-plus-ingress/tags/list --key --cert ``` ```json { "name": "nginx-ic/lts/nginx-plus-ingress", "tags": [ "" ] } ``` The `jq` command was used in these examples to make the JSON output easier to read. ### Add the image from source The [Installation with Helm](/nic/lts/install/helm/#install-the-helm-chart-from-source) documentation has a section describing how to use sources: these are the unique steps for Docker secrets using JWT tokens. 1. Clone the NGINX [`kubernetes-ingress` repository](https://github.com/nginx/kubernetes-ingress). 1. Navigate to the `charts/nginx-ingress` folder of your local clone. 1. Open the `values.yaml` file in an editor. You must change a few lines NGINX Ingress Controller LTS with NGINX Plus to be deployed. 1. Change the `nginxplus` argument to `true`. 1. Change the `repository` argument to the NGINX Ingress Controller LTS image you intend to use. 1. Add an argument to `imagePullSecretName` or `imagePullSecretsNames` to allow Docker to pull the image from the private registry. The following code block shows snippets of the parameters you will need to change, and an example of their contents: ```yaml ## Deploys the Ingress Controller for NGINX Plus nginxplus: true ## Truncated fields ## ... ## ... image: ## The image repository for the desired NGINX Ingress Controller LTS image repository: private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress ## The version tag tag: serviceAccount: ## The annotations of the service account of the Ingress Controller pods. annotations: {} ## Truncated fields ## ... ## ... ## The name of the secret containing docker registry credentials. ## Secret must exist in the same namespace as the helm release. ## Note that also imagePullSecretsNames can be used here if multiple secrets need to be set. imagePullSecretName: regcred ``` With the modified `values.yaml` file, you can now use Helm to install NGINX Ingress Controller LTS, for example: ```shell helm install nicdev01 -n nginx-ingress --create-namespace -f values.yaml . ``` The above command will install NGINX Ingress Controller LTS in the `nginx-ingress` namespace. If the namespace does not exist, `--create-namespace` will create it. Using `-f values.yaml` tells Helm to use the `values.yaml` file that you modified earlier with the settings you want to apply for your NGINX Ingress Controller LTS deployment. ## Manifest deployment The page ["Installation with Manifests"](/nic/lts/install/manifests.md) explains how to install NGINX Ingress Controller LTS using manifests. The following snippet is an example of a deployment: ```yaml spec: serviceAccountName: nginx-ingress imagePullSecrets: - name: regcred automountServiceAccountToken: true securityContext: seccompProfile: type: RuntimeDefault containers: - image: private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress: imagePullPolicy: IfNotPresent name: nginx-plus-ingress ``` The `imagePullSecrets` and `containers.image` lines represent the Kubernetes secret, as well as the registry and version of NGINX Ingress Controller LTS we are going to deploy. ## Download an image for local use If you need to download an image for local use (Such as to push to a different container registry), use this command: ```shell docker login private-registry.nginx.com --username= --password=none ``` Replace the contents of `` with the contents of the JWT token itself. Once you have successfully pulled the image, you can then tag it as needed. **Note:** For security, follow these practices with JSON Web Tokens (JWTs), passwords, and shell history: 1. **JWTs:** JWTs are sensitive information. Store them securely. Delete them after use to prevent unauthorized access. 1. **Shell history:** Commands that include JWTs or passwords are recorded in the history of your shell, in plain text. Clear your shell history after running such commands. For example, if you use bash, you can delete commands in your `~/.bash_history` file. Alternatively, you can run the `history -c` command to erase your shell history. Follow these practices to help ensure the security of your system and data. -------------------------------------------------------------------------------- title: "Advanced configuration with Annotations" url: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/index.md -------------------------------------------------------------------------------- This topic explains how to enable advanced features in F5 NGINX Ingress Controller with Annotations. The Ingress resource can use basic NGINX features such as host or path-based routing and TLS termination. Advanced features like rewriting the request URI or inserting additional response headers can be enabled with Annotations. Outside of advanced features, Annotations are necessary for customizing NGINX behavior such as setting the value of connection timeouts. Customization is also available through the [ConfigMap](/nic/configuration/global-configuration/configmap-resource.md) resources: Annotations take priority. ## Using Annotations This example uses Annotations to customize the configuration for an Ingress resource: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress-with-annotations annotations: nginx.org/proxy-connect-timeout: "30s" nginx.org/proxy-read-timeout: "20s" nginx.org/client-max-body-size: "4m" nginx.org/server-snippets: | location / { return 302 /coffee; } spec: rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` ## Validation NGINX Ingress Controller validates the annotations of Ingress resources. If an Ingress is invalid, NGINX Ingress Controller will reject it: the Ingress will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. You can check if NGINX Ingress Controller successfully applied the configuration for an Ingress resource. For the example `cafe-ingress-with-annotations` Ingress, you can run: ```shell kubectl describe ing cafe-ingress-with-annotations ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/cafe-ingress-with-annotations was added or updated ``` The events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid Ingress, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create an Ingress `cafe-ingress-with-annotations`, with an annotation `nginx.org/redirect-to-https` set to `yes please` instead of `true`, you will get: ```shell kubectl describe ing cafe-ingress-with-annotations ``` ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 13s nginx-ingress-controller annotations.nginx.org/redirect-to-https: Invalid value: "yes please": must be a boolean ``` Note how the events section includes a Warning event with the Rejected reason. **Note:** If you make an existing Ingress invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. The `nginx.com/jwt-token` Ingress annotation has limited validation. ## Summary of Annotations The table below summarizes the available annotations. **Note:** Annotations that start with `nginx.com` are only supported with NGINX Plus. For session persistence, use `nginx.org/sticky-cookie-services`, which works with both NGINX and NGINX Plus. ### General customization |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/proxy-connect-timeout* | *proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | | *nginx.org/proxy-read-timeout* | *proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | | *nginx.org/proxy-send-timeout* | *proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | | *nginx.org/proxy-next-upstream* | N/A | Sets the value of the [proxy_next_upstream](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) directive. | *error timeout* | *off* | | *nginx.org/proxy-next-upstream-timeout* | N/A | Sets the value of the [proxy_next_upstream_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. | *0s* | *3m* | | *nginx.org/proxy-next-upstream-tries* | N/A | Sets the value of the [proxy_next_upstream_tries](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. | *0* | *5* | | *nginx.org/client-max-body-size* | *client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | | *nginx.org/client-body-buffer-size* | *client-body-buffer-size* | Sets the value of the [client_body_buffer_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) directive. | N/A | | | *nginx.org/proxy-buffering* | *proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | | *nginx.org/proxy-buffers* | *proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | | *nginx.org/proxy-buffer-size* | *proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | | *nginx.org/proxy-busy-buffers-size* | *proxy-busy-buffers-size* | Sets the value of the [proxy_busy_buffers_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) directive. | Depends on the platform. | | | *nginx.org/proxy-max-temp-file-size* | *proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | | *nginx.org/server-tokens* | *server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. Additionally, with the NGINX Plus, you can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | | *nginx.org/path-regex* | N/A | Enables regular expression modifiers for Ingress path parameter. This translates to the NGINX [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive. You can specify one of these values: "case_sensitive", "case_insensitive", or "exact". The annotation is applied to the entire Ingress resource and its paths. While using Master and Minion Ingresses i.e. Mergeable Ingresses, this annotation can be specified on Minion types. The `path-regex` annotation specified on Master is ignored, and has no effect on paths defined on Minions. | N/A | [path-regex](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/path-regex) | | *nginx.org/policies* | N/A | Applies one or more [Policy resources](/nic/configuration/policy-resource/before-you-begin.md) to an Ingress. Specify a comma-separated list of policy names. | N/A | *webapp-policy* | | *nginx.com/policies* | N/A | Applies one or more [Policy resources](/nic/configuration/policy-resource/before-you-begin.md) to an Ingress enabling NGINX Plus use-cases. Specify a comma-separated list of policy names. | N/A | *waf-policy* | ### URI and Header Manipulation |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/proxy-hide-headers* | *proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: `"nginx.org/proxy-hide-headers": "header-a,header-b"` | N/A | | | *nginx.org/proxy-pass-headers* | *proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: `"nginx.org/proxy-pass-headers": "header-a,header-b"` | N/A | | | *nginx.org/rewrites* | N/A | Configures URI rewriting using [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive. | N/A | [rewrites](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/rewrites) | | *nginx.org/rewrite-target* | N/A | Configures URI rewriting using the [rewrite](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite) directive. The annotation value specifies the target path that requests should be rewritten to. Supports regex capture groups (`$1`, `$2`, etc.) when used with `nginx.org/path-regex`. Mutually exclusive with `nginx.org/rewrites`. | N/A | [rewrite-target](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/rewrite-target) | |*nginx.org/proxy-set-headers* | N/A | Enables customization of proxy headers and values using the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. Example: `"nginx.org/proxy-set-headers": "header-a: valueA,header-b: valueB,header-c: valueC"`. `Host` is not supported; use `nginx.org/upstream-vhost`. | N/A | [Proxy Set Headers](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/proxy-set-headers). | | *nginx.org/upstream-vhost* | N/A | Sets the `Host` header sent to the upstream to a fixed hostname, using the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive (or [grpc_set_header](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_set_header) for gRPC backends). When not set, the upstream receives the Host header from the original request. Only a literal hostname is accepted; NGINX variables are not supported. | N/A | *example.internal* [upstream-vhost](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/upstream-vhost) | | *nginx.org/add-header* | *add-header* | Adds one or more response headers with the [add_header](https://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header) directive. Use the format `Header-Name: value[:always]` and separate entries with commas. | N/A | *X-Frame-Options: DENY: always, X-Content-Type-Options: nosniff* | | *nginx.org/add-header-inherit* | *add-header-inherit* | Controls how [add_header_inherit](https://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header_inherit) applies inherited response headers. Allowed values are `on`, `off`, and `merge`. | N/A | *merge* | | *nginx.org/proxy-redirect-from* | N/A | The `redirect` parameter for [proxy_redirect](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect). Accepts `off`, `default`, a URL string, or a regex prefixed with `~` (case-sensitive) or `~*` (case-insensitive). Requires `nginx.org/proxy-redirect-to` when value is a URL or regex. | N/A | *http://redirect-backend-svc/v1/* or *off* | | *nginx.org/proxy-redirect-to* | N/A | The `replacement` parameter for [proxy_redirect](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect). Required when `nginx.org/proxy-redirect-from` is a URL or regex; must not be set without `nginx.org/proxy-redirect-from`. | N/A | *http://cafe.example.com/coffee/* | ### Auth and SSL/TLS |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/app-root* | N/A | Configures the application root path that the controller redirects requests for / to. Returns 302 redirect that will take precedence over other redirects. | N/A | `/` redirects to `/coffee` | | *nginx.org/redirect-to-https* | *redirect-to-https* | Sets a redirect rule based on the value of the `http_x_forwarded_proto` header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of NGINX Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115). The redirect code can be configured with the `nginx.org/http-redirect-code` annotation or the `http-redirect-code` ConfigMap key. | *False* | | | *nginx.org/ssl-redirect* | *ssl-redirect* | Sets a redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS when TLS is configured. The redirect code can be configured with the `nginx.org/http-redirect-code` annotation or the `http-redirect-code` ConfigMap key. | *True* | | | *nginx.org/http-redirect-code* | *http-redirect-code* | Sets the HTTP redirect code for HTTPS redirects. Supported codes: 301, 302, 307, 308. | *301* | *307* | | *nginx.org/hsts* | *hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/)\ : the HSTS header is added to the responses from backends. The `preload` directive is included in the header. | *False* | | | *nginx.org/hsts-max-age* | *hsts-max-age* | Sets the value of the `max-age` directive of the HSTS header. | *2592000* (1 month) | | | *nginx.org/hsts-include-subdomains* | *hsts-include-subdomains* | Adds the `includeSubDomains` directive to the HSTS header. | *False* | | | *nginx.org/hsts-behind-proxy* | *hsts-behind-proxy* | Enables HSTS based on the value of the `http_x_forwarded_proto` request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of NGINX Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the `nginx.org/redirect-to-https` annotation. | *False* | | | *nginx.org/basic-auth-secret* | N/A | Specifies a Secret resource with a user list for HTTP Basic authentication. | N/A | | | *nginx.org/basic-auth-realm* | N/A | Specifies a realm. | N/A | | | *nginx.org/ssl-ciphers* | *ssl-ciphers* | Sets the value of the [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive. | N/A | *HIGH:!aNULL:!MD5* | | *nginx.org/ssl-prefer-server-ciphers* | *ssl-prefer-server-ciphers* | Enables or disables the [ssl_prefer_server_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) directive. | *False* | | | *nginx.com/jwt-key* | N/A | Specifies a Secret resource with keys for validating JSON Web Tokens (JWTs). | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/jwt). | | *nginx.com/jwt-realm* | N/A | Specifies a realm. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/jwt). | | *nginx.com/jwt-token* | N/A | Specifies a variable that contains a JSON Web Token. | By default, a JWT is expected in the `Authorization` header as a Bearer Token. | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/jwt). | | *nginx.com/jwt-login-url* | N/A | Specifies a URL to which a client is redirected in case of an invalid or missing JWT. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/jwt). | ### Listeners | Annotation | ConfigMap Key | Description | Default | | ---| ---| ---| ---| | *nginx.org/listen-ports* | N/A | Configures HTTP ports that NGINX will listen on. Not supported on Ingress resources without a `host`; use the `-default-http-listener-port` command-line argument instead. | *[80]* | | *nginx.org/listen-ports-ssl* | N/A | Configures HTTPS ports that NGINX will listen on. Not supported on Ingress resources without a `host`; use the `-default-https-listener-port` command-line argument instead. | *[443]* | ### Backend services (Upstreams) |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/lb-method* | *lb-method* | Sets the [load balancing method](/nginx/admin-guide/load-balancer/http-load-balancer.md#choosing-a-load-balancing-method). To use the round-robin method, specify `"round_robin"`. | `"random two least_conn"` | | | *nginx.org/ssl-services* | N/A | Enables HTTPS or gRPC over SSL when connecting to the endpoints of services. | N/A | [ssl-services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/ssl-services) | | *nginx.org/grpc-services* | N/A | Enables gRPC for services. Note: requires HTTP/2 (see `http2` ConfigMap key); only works for Ingresses with TLS termination enabled. | N/A | [grpc-services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/grpc-services) | | *nginx.org/websocket-services* | N/A | Enables WebSocket for services. | N/A | [websocket](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/websocket) | | *nginx.org/max-fails* | *max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the `server` directive. | *1* | | | *nginx.org/max-conns* | N\A | Sets the value of the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the `server` directive. | *0* | | | *nginx.org/upstream-zone-size* | *upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. For NGINX, the special value 0 disables the shared memory zones. For NGINX Plus, shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256K* | | | *nginx.org/fail-timeout* | *fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the `server` directive. | *10s* | | | *nginx.org/sticky-cookie-services* | N/A | Configures session persistence. This is the preferred annotation for session persistence configuration and works with both NGINX (since version 1.29.6) and NGINX Plus. If both `nginx.org/sticky-cookie-services` and `nginx.com/sticky-cookie-services` are set, the `nginx.org/` annotation takes precedence. *Requires NGINX >= v1.29.6*. | N/A | [session-persistence](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/session-persistence) | | *nginx.com/sticky-cookie-services* | N/A | Configures session persistence. Maintained for backward compatibility with NGINX Plus. Use `nginx.org/sticky-cookie-services` instead. | N/A | [session-persistence](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/session-persistence) | | *nginx.org/keepalive* | *keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that `proxy_set_header Connection "";` is added to the generated configuration when the value > 0. | *0* | | | *nginx.com/health-checks* | N/A | Enables active health checks. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/health-checks) | | *nginx.com/health-checks-mandatory* | N/A | Configures active health checks as mandatory. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/health-checks) | | *nginx.com/health-checks-mandatory-queue* | N/A | When active health checks are mandatory, creates a queue where incoming requests are temporarily stored while NGINX Plus is checking the health of the endpoints after a configuration reload. | *0* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/health-checks) | | *nginx.com/slow-start* | N/A | Sets the upstream server [slow-start period](/nginx/admin-guide/load-balancer/http-load-balancer.md#server-slow-start). By default, slow-start is activated after a server becomes [available](/nginx/admin-guide/load-balancer/http-health-check.md#passive-health-checks) or [healthy](/nginx/admin-guide/load-balancer/http-health-check.md#active-health-checks). To enable slow-start for newly-added servers, configure [mandatory active health checks](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/health-checks). | *"0s"* | | | *nginx.org/use-cluster-ip* | N/A | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like `lb-method` and `next-upstream`) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | *False* | | | *nginx.org/custom-http-errors* | N/A | Enables intercepting of backend responses using [proxy_intercept_errors](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_intercept_errors) and [error_page](https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page) directives with matching status codes and routes the intercepted request to the Ingress's `spec.defaultBackend` so a user-provided Service can render a custom error body, while preserving the original upstream status code. | N/A | [custom-http-errors](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/custom-http-errors) | ### Rate limiting |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/limit-req-rate* | N/A | Enables request-rate-limiting for this ingress by creating a [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) and matching [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) for each location. All servers/locations of one ingress share the same zone. Must have unit r/s or r/m. | N/A | 200r/s | | *nginx.org/limit-req-key* | N/A | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ${}. | ${binary_remote_addr} | ${binary_remote_addr} | | *nginx.org/limit-req-zone-size* | N/A | Configures the size of the created [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone). | 10m | 20m | | *nginx.org/limit-req-delay* | N/A | Configures the delay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | 0 | 100 | | *nginx.org/limit-req-no-delay* | N/A | Configures the nodelay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | false | true | | *nginx.org/limit-req-burst* | N/A | Configures the burst-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | N/A | 100 | | *nginx.org/limit-req-dry-run* | N/A | Enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone. | false | true | | *nginx.org/limit-req-log-level* | N/A | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are info, notice, warn or error. | error | info | | *nginx.org/limit-req-reject-code* | N/A | Sets the status code to return in response to rejected requests. Must fall into the range 400..599. | 429 | 503 | | *nginx.org/limit-req-scale* | N/A | Enables a constant rate-limit by dividing the configured rate by the number of nginx-ingress pods currently serving traffic. This adjustment ensures that the rate-limit remains consistent, even as the number of nginx-pods fluctuates due to autoscaling. Note: This will not work properly if requests from a client are not evenly distributed accross all ingress pods (sticky sessions, long lived TCP-Connections with many requests etc.). In such cases using [zone-sync](/nic/configuration/global-configuration/configmap-resource.md#zone-sync) instead would give better results. Enabling `zone-sync` will suppress this setting. | false | true | ### Snippets and custom templates | Annotation | ConfigMap Key | Description | Default | | ---| ---| ---| ---| | *nginx.org/location-snippets* | *location-snippets* | Sets a custom snippet in location context. | N/A | | *nginx.org/server-snippets* | *server-snippets* | Sets a custom snippet in server context. | N/A | ### F5 WAF for NGINX {#app-protect} **Note:** The App Protect annotations only work if the F5 WAF for NGINX module is [installed](/nic/integrations/app-protect-waf/installation.md). |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *appprotect.f5.com/app-protect-policy* | N/A | The name of the App Protect Policy for the Ingress Resource. Format is `namespace/name`. If no namespace is specified, the same namespace of the Ingress Resource is used. If not specified but `appprotect.f5.com/app-protect-enable` is true, a default policy id applied. If the referenced policy resource does not exist, or policy is invalid, this annotation will be ignored, and the default policy will be applied. | N/A | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-waf) | | *appprotect.f5.com/app-protect-enable* | N/A | Enable App Protect for the Ingress Resource. | *False* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-waf) | | *appprotect.f5.com/app-protect-security-log-enable* | N/A | Enable the [security log](/nginx-app-protect/troubleshooting/#app-protect-logging-overview) for App Protect. | *False* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-waf) | | *appprotect.f5.com/app-protect-security-log* | N/A | The App Protect log configuration for the Ingress Resource. Format is `namespace/name`. If no namespace is specified, the same namespace as the Ingress Resource is used. If not specified the default is used which is: filter: `illegal`, format: `default`. Multiple configurations can be specified in a comma separated list. Both log configurations and destinations list (see below) must be of equal length. Configs and destinations are paired by the list indices. | N/A | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-waf) | | *appprotect.f5.com/app-protect-security-log-destination* | N/A | The destination of the security log. For more information check the [DESTINATION argument](/nginx-app-protect/troubleshooting/#app-protect-logging-overview). Multiple destinations can be specified in a comma-separated list. Both log configurations and destinations list (see above) must be of equal length. Configs and destinations are paired by the list indices. | *syslog:server=localhost:514* | [app-protect-waf](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-waf) | ### App Protect DoS **Note:** The App Protect DoS annotations only work if the App Protect DoS module is [installed](/nic/integrations/app-protect-dos/installation.md). |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *appprotectdos.f5.com/app-protect-dos-resource* | N/A | Enable App Protect DoS for the Ingress Resource by specifying a [DosProtectedResource](/nic/integrations/app-protect-dos/dos-protected.md). | N/A | [app-protect-dos](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-dos) | -------------------------------------------------------------------------------- title: "Advanced configuration with Annotations" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/ingress-resources/advanced-configuration-with-annotations/index.md -------------------------------------------------------------------------------- This topic explains how to enable advanced features in F5 NGINX Ingress Controller LTS with Annotations. The Ingress resource can use basic NGINX features such as host or path-based routing and TLS termination. Advanced features like rewriting the request URI or inserting additional response headers can be enabled with Annotations. Outside of advanced features, Annotations are necessary for customizing NGINX behavior such as setting the value of connection timeouts. Customization is also available through the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource.md) resources: Annotations take priority. ## Using Annotations This example uses Annotations to customize the configuration for an Ingress resource: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress-with-annotations annotations: nginx.org/proxy-connect-timeout: "30s" nginx.org/proxy-read-timeout: "20s" nginx.org/client-max-body-size: "4m" nginx.org/server-snippets: | location / { return 302 /coffee; } spec: rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` ## Validation NGINX Ingress Controller LTS validates the annotations of Ingress resources. If an Ingress is invalid, NGINX Ingress Controller LTS will reject it: the Ingress will continue to exist in the cluster, but NGINX Ingress Controller LTS will ignore it. You can check if NGINX Ingress Controller LTS successfully applied the configuration for an Ingress resource. For the example `cafe-ingress-with-annotations` Ingress, you can run: ```shell kubectl describe ing cafe-ingress-with-annotations ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/cafe-ingress-with-annotations was added or updated ``` The events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid Ingress, NGINX Ingress Controller LTS will reject it and emit a Rejected event. For example, if you create an Ingress `cafe-ingress-with-annotations`, with an annotation `nginx.org/redirect-to-https` set to `yes please` instead of `true`, you will get: ```shell kubectl describe ing cafe-ingress-with-annotations ``` ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 13s nginx-ingress-controller annotations.nginx.org/redirect-to-https: Invalid value: "yes please": must be a boolean ``` Note how the events section includes a Warning event with the Rejected reason. **note:** If you make an existing Ingress invalid, NGINX Ingress Controller LTS will reject it and remove the corresponding configuration from NGINX. The `nginx.com/jwt-token` Ingress annotation has limited validation. ## Summary of Annotations The table below summarizes the available annotations. ### General customization |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/proxy-connect-timeout* | *proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | | *nginx.org/proxy-read-timeout* | *proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | | *nginx.org/proxy-send-timeout* | *proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | | *nginx.org/proxy-next-upstream* | N/A | Sets the value of the [proxy_next_upstream](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) directive. | *error timeout* | *off* | | *nginx.org/proxy-next-upstream-timeout* | N/A | Sets the value of the [proxy_next_upstream_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. | *0s* | *3m* | | *nginx.org/proxy-next-upstream-tries* | N/A | Sets the value of the [proxy_next_upstream_tries](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. | *0* | *5* | | *nginx.org/client-max-body-size* | *client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | | *nginx.org/client-body-buffer-size* | *client-body-buffer-size* | Sets the value of the [client_body_buffer_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) directive. | N/A | | | *nginx.org/proxy-buffering* | *proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | | *nginx.org/proxy-buffers* | *proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | | *nginx.org/proxy-buffer-size* | *proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | | *nginx.org/proxy-busy-buffers-size* | *proxy-busy-buffers-size* | Sets the value of the [proxy_busy_buffers_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) directive. | Depends on the platform. | | | *nginx.org/proxy-max-temp-file-size* | *proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | | *nginx.org/server-tokens* | *server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. You can also specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | | *nginx.org/path-regex* | N/A | Enables regular expression modifiers for Ingress path parameter. This translates to the NGINX [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive. You can specify one of these values: "case_sensitive", "case_insensitive", or "exact". The annotation is applied to the entire Ingress resource and its paths. While using Master and Minion Ingresses i.e. Mergeable Ingresses, this annotation can be specified on Minion types. The `path-regex` annotation specified on Master is ignored, and has no effect on paths defined on Minions. | N/A | [path-regex](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/path-regex) | | *nginx.org/policies* | N/A | Applies one or more [Policy resources](/nic/lts/configuration/policy-resource/before-you-begin.md) to an Ingress. Specify a comma-separated list of policy names. | N/A | *webapp-policy* | ### Request URI/Header Manipulation |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/proxy-hide-headers* | *proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: `"nginx.org/proxy-hide-headers": "header-a,header-b"` | N/A | | | *nginx.org/proxy-pass-headers* | *proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: `"nginx.org/proxy-pass-headers": "header-a,header-b"` | N/A | | | *nginx.org/rewrites* | N/A | Configures URI rewriting using [proxy_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive. | N/A | [rewrites](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/rewrites) | | *nginx.org/rewrite-target* | N/A | Configures URI rewriting using the [rewrite](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite) directive. The annotation value specifies the target path that requests should be rewritten to. Supports regex capture groups (`$1`, `$2`, etc.) when used with `nginx.org/path-regex`. Mutually exclusive with `nginx.org/rewrites`. | N/A | [rewrite-target](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/rewrite-target) | |*nginx.org/proxy-set-headers* | N/A | Enables customization of proxy headers and values using the [proxy_set_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive. Example: `"nginx.org/proxy-set-headers": "header-a: valueA,header-b: valueB,header-c: valueC"` | N/A | [Proxy Set Headers](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/proxy-set-headers). | ### Auth and SSL/TLS |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/app-root* | N/A | Configures the application root path that the controller redirects requests for / to. Returns 302 redirect that will take precedence over other redirects. | N/A | `/` redirects to `/coffee` | | *nginx.org/redirect-to-https* | *redirect-to-https* | Sets a redirect rule based on the value of the `http_x_forwarded_proto` header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of NGINX Ingress Controller LTS — see [115](https://github.com/nginx/kubernetes-ingress/issues/115). The redirect code can be configured with the `nginx.org/http-redirect-code` annotation or the `http-redirect-code` ConfigMap key. | *False* | | | *nginx.org/ssl-redirect* | *ssl-redirect* | Sets a redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS when TLS is configured. The redirect code can be configured with the `nginx.org/http-redirect-code` annotation or the `http-redirect-code` ConfigMap key. | *True* | | | *nginx.org/http-redirect-code* | *http-redirect-code* | Sets the HTTP redirect code for HTTPS redirects. Supported codes: 301, 302, 307, 308. | *301* | *307* | | *nginx.org/hsts* | *hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/)\ : the HSTS header is added to the responses from backends. The `preload` directive is included in the header. | *False* | | | *nginx.org/hsts-max-age* | *hsts-max-age* | Sets the value of the `max-age` directive of the HSTS header. | *2592000* (1 month) | | | *nginx.org/hsts-include-subdomains* | *hsts-include-subdomains* | Adds the `includeSubDomains` directive to the HSTS header. | *False* | | | *nginx.org/hsts-behind-proxy* | *hsts-behind-proxy* | Enables HSTS based on the value of the `http_x_forwarded_proto` request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of NGINX Ingress Controller LTS. Note: to control redirection from HTTP to HTTPS configure the `nginx.org/redirect-to-https` annotation. | *False* | | | *nginx.org/basic-auth-secret* | N/A | Specifies a Secret resource with a user list for HTTP Basic authentication. | N/A | | | *nginx.org/basic-auth-realm* | N/A | Specifies a realm. | N/A | | | *nginx.org/ssl-ciphers* | *ssl-ciphers* | Sets the value of the [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive. | N/A | *HIGH:!aNULL:!MD5* | | *nginx.org/ssl-prefer-server-ciphers* | *ssl-prefer-server-ciphers* | Enables or disables the [ssl_prefer_server_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) directive. | *False* | | | *nginx.com/jwt-key* | N/A | Specifies a Secret resource with keys for validating JSON Web Tokens (JWTs). | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/jwt). | | *nginx.com/jwt-realm* | N/A | Specifies a realm. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/jwt). | | *nginx.com/jwt-token* | N/A | Specifies a variable that contains a JSON Web Token. | By default, a JWT is expected in the `Authorization` header as a Bearer Token. | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/jwt). | | *nginx.com/jwt-login-url* | N/A | Specifies a URL to which a client is redirected in case of an invalid or missing JWT. | N/A | [Support for JSON Web Tokens (JWTs)](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/jwt). | ### Listeners | Annotation | ConfigMap Key | Description | Default | | ---| ---| ---| ---| | *nginx.org/listen-ports* | N/A | Configures HTTP ports that NGINX will listen on. | *[80]* | | *nginx.org/listen-ports-ssl* | N/A | Configures HTTPS ports that NGINX will listen on. | *[443]* | ### Backend services (Upstreams) |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/lb-method* | *lb-method* | Sets the [load balancing method](/nginx/admin-guide/load-balancer/http-load-balancer.md#choosing-a-load-balancing-method). To use the round-robin method, specify `"round_robin"`. | `"random two least_conn"` | | | *nginx.org/ssl-services* | N/A | Enables HTTPS or gRPC over SSL when connecting to the endpoints of services. | N/A | [ssl-services](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/ssl-services) | | *nginx.org/grpc-services* | N/A | Enables gRPC for services. Note: requires HTTP/2 (see `http2` ConfigMap key); only works for Ingresses with TLS termination enabled. | N/A | [grpc-services](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/grpc-services) | | *nginx.org/websocket-services* | N/A | Enables WebSocket for services. | N/A | [websocket](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/websocket) | | *nginx.org/max-fails* | *max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the `server` directive. | *1* | | | *nginx.org/max-conns* | N\A | Sets the value of the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the `server` directive. | *0* | | | *nginx.org/upstream-zone-size* | *upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. Shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256K* | | | *nginx.org/fail-timeout* | *fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the `server` directive. | *10s* | | | *nginx.org/sticky-cookie-services* | N/A | Configures session persistence. This is the preferred annotation for session persistence configuration. If both `nginx.org/sticky-cookie-services` and `nginx.com/sticky-cookie-services` are set, the `nginx.org/` annotation takes precedence. | N/A | [session-persistence](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/session-persistence) | | *nginx.com/sticky-cookie-services* | N/A | Configures session persistence. Maintained for backward compatibility. Use `nginx.org/sticky-cookie-services` instead. | N/A | [session-persistence](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/session-persistence) | | *nginx.org/keepalive* | *keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that `proxy_set_header Connection "";` is added to the generated configuration when the value > 0. | *0* | | | *nginx.com/health-checks* | N/A | Enables active health checks. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/health-checks) | | *nginx.com/health-checks-mandatory* | N/A | Configures active health checks as mandatory. | *False* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/health-checks) | | *nginx.com/health-checks-mandatory-queue* | N/A | When active health checks are mandatory, creates a queue where incoming requests are temporarily stored while NGINX Plus is checking the health of the endpoints after a configuration reload. | *0* | [health-checks](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/health-checks) | | *nginx.com/slow-start* | N/A | Sets the upstream server [slow-start period](/nginx/admin-guide/load-balancer/http-load-balancer.md#server-slow-start). By default, slow-start is activated after a server becomes [available](/nginx/admin-guide/load-balancer/http-health-check.md#passive-health-checks) or [healthy](/nginx/admin-guide/load-balancer/http-health-check.md#active-health-checks). To enable slow-start for newly-added servers, configure [mandatory active health checks](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/health-checks). | *"0s"* | | | *nginx.org/use-cluster-ip* | N/A | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like `lb-method` and `next-upstream`) will have no effect, as NGINX Ingress Controller LTS will configure NGINX with only one upstream server that will match the service Cluster IP. | *False* | | ### Rate limiting |Annotation | ConfigMap Key | Description | Default | Example | | ---| ---| ---| ---| --- | | *nginx.org/limit-req-rate* | N/A | Enables request-rate-limiting for this ingress by creating a [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) and matching [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) for each location. All servers/locations of one ingress share the same zone. Must have unit r/s or r/m. | N/A | 200r/s | | *nginx.org/limit-req-key* | N/A | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ${}. | ${binary_remote_addr} | ${binary_remote_addr} | | *nginx.org/limit-req-zone-size* | N/A | Configures the size of the created [limit_req_zone](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone). | 10m | 20m | | *nginx.org/limit-req-delay* | N/A | Configures the delay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | 0 | 100 | | *nginx.org/limit-req-no-delay* | N/A | Configures the nodelay-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | false | true | | *nginx.org/limit-req-burst* | N/A | Configures the burst-parameter of the [limit_req](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req) directive. | N/A | 100 | | *nginx.org/limit-req-dry-run* | N/A | Enables the dry run mode. In this mode, the rate limit is not actually applied, but the number of excessive requests is accounted as usual in the shared memory zone. | false | true | | *nginx.org/limit-req-log-level* | N/A | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are info, notice, warn or error. | error | info | | *nginx.org/limit-req-reject-code* | N/A | Sets the status code to return in response to rejected requests. Must fall into the range 400..599. | 429 | 503 | | *nginx.org/limit-req-scale* | N/A | Enables a constant rate-limit by dividing the configured rate by the number of nginx-ingress pods currently serving traffic. This adjustment ensures that the rate-limit remains consistent, even as the number of nginx-pods fluctuates due to autoscaling. Note: This will not work properly if requests from a client are not evenly distributed accross all ingress pods (sticky sessions, long lived TCP-Connections with many requests etc.). In such cases using [zone-sync](/nic/lts/configuration/global-configuration/configmap-resource.md#zone-sync) instead would give better results. Enabling `zone-sync` will suppress this setting. | false | true | ### Snippets and custom templates | Annotation | ConfigMap Key | Description | Default | | ---| ---| ---| ---| | *nginx.org/location-snippets* | *location-snippets* | Sets a custom snippet in location context. | N/A | | *nginx.org/server-snippets* | *server-snippets* | Sets a custom snippet in server context. | N/A | -------------------------------------------------------------------------------- title: "Configuration" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-dos/configuration/index.md -------------------------------------------------------------------------------- **Note:** Check out the complete [NGINX Ingress Controller with App Protect DoS example for VirtualServer](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/app-protect-dos) and the [NGINX Ingress Controller with App Protect DoS example for Ingress](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/app-protect-dos). ## App Protect DoS Configuration A `DosProtectedResource` is a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) that holds the configuration of a collection of protected resources. An [Ingress](/nic/configuration/ingress-resources/basic-configuration), [VirtualServer and VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) can be protected by specifying a reference to the DosProtectedResource. 1. Create an `DosProtectedResource` Custom resource manifest. As an example: ```yaml apiVersion: appprotectdos.f5.com/v1beta1 kind: DosProtectedResource metadata: name: dos-protected spec: enable: true name: "webapp.example.com" apDosMonitor: uri: "webapp.example.com" protocol: "http1" timeout: 5 ``` 2. Enable App Protect DoS for an Ingress resource by adding an annotation to the Ingress. Set the value of the annotation to the qualified identifier(`namespace/name`) of a DosProtectedResource: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webapp-ingress annotations: appprotectdos.f5.com/app-protect-dos-resource: "default/dos-protected" ``` 3. Enable App Protect DoS on a VirtualServer resource by setting the `dos` field value to the qualified identifier(`namespace/name`) of a DosProtectedResource: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: webapp spec: host: webapp.example.com upstreams: - name: webapp service: webapp-svc port: 80 routes: - path: / dos: dos-protected action: pass: webapp ``` ## DoS Policy Configuration You can configure the policy for DoS by creating an `APDosPolicy` [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and specifying the qualified identifier(`namespace/name`) of the `ApDosPolicy` in the `DosProtectedResource`. For example, say you want to use DoS Policy as shown below: ```json { mitigation_mode: "standard", signatures: "on", bad_actors: "on", automation_tools_detection: "on", tls_fingerprint: "on", } ``` Create an `APDosPolicy` resource with the policy defined in the `spec`, as shown below: ```yaml apiVersion: appprotectdos.f5.com/v1beta1 kind: APDosPolicy metadata: name: dospolicy spec: mitigation_mode: "standard" signatures: "on" bad_actors: "on" automation_tools_detection: "on" tls_fingerprint: "on" ``` Then add a reference in the `DosProtectedResource` to the `ApDosPolicy`: ```yaml apiVersion: appprotectdos.f5.com/v1beta1 kind: DosProtectedResource metadata: name: dos-protected spec: enable: true name: "my-dos" apDosMonitor: uri: "webapp.example.com" apDosPolicy: "default/dospolicy" ``` ## App Protect DoS Logs {#app-protect-dos-logs} You can set the [App Protect DoS Log configuration](/nap-dos/monitoring/types-of-logs.md) by creating an `APDosLogConf` [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and specifying the qualified identifier(`namespace/name`) of the `ApDosLogConf` in the `DosProtectedResource`. For example, say you want to log state changing requests for your Ingress resources using App Protect DoS. The App Protect DoS log configuration looks like this: ```json { "filter": { "traffic-mitigation-stats": "all", "bad-actors": "top 10", "attack-signatures": "top 10" } } ``` You would add that config in the `spec` of your `APDosLogConf` resource as follows: ```yaml apiVersion: appprotectdos.f5.com/v1beta1 kind: APDosLogConf metadata: name: doslogconf spec: filter: traffic-mitigation-stats: all bad-actors: top 10 attack-signatures: top 10 ``` Then add a reference in the `DosProtectedResource` to the `APDosLogConf`: ```yaml apiVersion: appprotectdos.f5.com/v1beta1 kind: DosProtectedResource metadata: name: dos-protected spec: enable: true name: "my-dos" apDosMonitor: uri: "webapp.example.com" dosSecurityLog: enable: true apDosLogConf: "doslogconf" dosLogDest: "syslog-svc.default.svc.cluster.local:514" ``` ## Global Configuration NGINX Ingress Controller has a set of global configuration parameters that align with those available in the F5 DoS for NGINX module. See [ConfigMap keys](/nic/configuration/global-configuration/configmap-resource.md#modules) for the complete list. The App Protect parameters use the `app-protect-dos*` prefix. -------------------------------------------------------------------------------- title: "Configure F5 WAF for NGINX with NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-waf-v5/configuration/index.md -------------------------------------------------------------------------------- ## Overview This document explains how to use F5 NGINX Ingress Controller to configure [F5 WAF for NGINX v5](/waf/). **Note:** There are complete NGINX Ingress Controller with F5 WAF for NGINX [example resources on GitHub](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/app-protect-waf-v5). ## Global configuration NGINX Ingress Controller has global configuration parameters that match those in F5 WAF for NGINX. They are found in the [ConfigMap resource](/nic/configuration/global-configuration/configmap-resource.md#modules): the F5 WAF for NGINX parameters are prefixed with `app-protect*`. ## Enable F5 WAF for NGINX v5 F5 WAF for NGINX v5 can be enabled for VirtualServer, VirtualServerRoute, and Ingress resources through a Policy resource that references a WAF bundle or bundle source. Additional detail can be found in the [Policy Resource documentation](/nic/configuration/policy-resource/policy-reference.md#waf). **Note:** Policies that rely on NGINX subrequests (such as `ExternalAuth`, `APIKey`, `JWT` with remote JWKS fetching, `OIDC`, or `Cache` with `cacheBackgroundUpdate`) and F5 WAF for NGINX may not function as expected and may cause issues when applied together on the same route. --- ## F5 WAF for NGINX Bundles F5 WAF for NGINX bundles for VirtualServer custom resources are defined by creating policy bundles and putting them on a mounted volume accessible from NGINX Ingress Controller. Before applying a policy, a WAF policy bundle must be created, then copied to a volume mounted to `/etc/app_protect/bundles`. **Note:** Instead of manually placing bundles on a mounted volume, you can configure NGINX Ingress Controller to fetch bundles from a remote source automatically. See [Connect F5 WAF for NGINX to bundle sources](/nic/integrations/app-protect-waf-v5/bundle-sources.md) for details on NGINX One Console, NGINX Instance Manager, and HTTPS source types. **Note:** NGINX Ingress Controller supports `securityLogs` for policy bundles. Log bundles must also be copied to a volume mounted to `/etc/app_protect/bundles`. This example shows how a policy is configured by referencing a generated WAF Policy Bundle: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: spec: waf: enable: true apBundle: ".tgz" ``` This example shows the same policy as above but with a log bundle used for security log configuration: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: spec: waf: enable: true apBundle: ".tgz" securityLogs: - enable: true apLogBundle: ".tgz" logDest: "syslog:server=syslog-svc.default:514" ``` --- ## Configure NGINX Plus Ingress Controller using Virtual Server resources This example shows how to deploy NGINX Ingress Controller with NGINX Plus and F5 WAF for NGINX v5, deploy a simple web application, and then configure load balancing and WAF protection for that application using the VirtualServer resource. **Note:** You can find the files for this example on [GitHub](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/app-protect-waf-v5). ### Prerequisites 1. Follow the installation [instructions](/nic/integrations/app-protect-waf-v5/installation.md) to deploy NGINX Ingress Controller with NGINX Plus and F5 WAF for NGINX version 5. 2. Save the public IP address of NGINX Ingress Controller into a shell variable: ```shell IC_IP=XXX.YYY.ZZZ.III ``` 3. Save the HTTP port of NGINX Ingress Controller into a shell variable: ```shell IC_HTTP_PORT= ``` ### Deploy a web application Create the application deployment and service: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/examples/custom-resources/app-protect-waf-v5/webapp.yaml ``` ### Create the Syslog service Create the syslog service and pod for the F5 WAF for NGINX security logs: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/examples/custom-resources/app-protect-waf-v5/syslog.yaml ``` ### Deploy the WAF Policy **Note:** Configuration settings in the Policy resource enable WAF protection by configuring F5 WAF for NGINX with the log configuration created in the previous step. The policy bundle referenced as `your_policy_bundle_name.tgz` need to be created and placed in the `/etc/app_protect/bundles` volume first. Create and deploy the WAF policy. ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/examples/custom-resources/app-protect-waf-v5/waf.yaml ``` ### Configure load balancing **Note:** VirtualServer references the `waf-policy` created in Step 3. 1. Create the VirtualServer Resource: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/examples/custom-resources/app-protect-waf-v5/virtual-server.yaml ``` ### Test the application To access the application, curl the coffee and the tea services. Use the `--resolve` option to set the Host header of a request with `webapp.example.com` 1. Send a request to the application: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT/ ``` ```shell Server address: 10.12.0.18:80 Server name: webapp-7586895968-r26zn ``` 1. Try to send a request with a suspicious URL: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP "http://webapp.example.com:$IC_HTTP_PORT/" ``` Expected output: ```text Request Rejected The requested URL was rejected. Please consult with your administrator. ... ``` ## Attach to an Ingress Save the following as `cafe.yaml`. The file defines the sample cafe application (`coffee` and `tea` Deployments and Services) and an Ingress. The `nginx.com/policies` annotation attaches the `waf-policy` Policy to every route on the Ingress. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: coffee namespace: default spec: replicas: 2 selector: matchLabels: app: coffee template: metadata: labels: app: coffee spec: containers: - name: coffee image: nginxdemos/hello:plain-text ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: coffee-svc namespace: default spec: selector: app: coffee ports: - port: 80 targetPort: 80 name: http --- apiVersion: apps/v1 kind: Deployment metadata: name: tea namespace: default spec: replicas: 2 selector: matchLabels: app: tea template: metadata: labels: app: tea spec: containers: - name: tea image: nginxdemos/hello:plain-text ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: tea-svc namespace: default spec: selector: app: tea ports: - port: 80 targetPort: 80 name: http --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress namespace: default annotations: nginx.com/policies: "waf-policy" spec: ingressClassName: nginx rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` Apply the file: ```shell kubectl apply -f cafe.yaml ``` Send a normal request: ```shell curl --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP \ http://cafe.example.com:$IC_HTTP_PORT/coffee ``` Send a request that triggers the data guard violation: ```shell curl --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP \ "http://cafe.example.com:$IC_HTTP_PORT/coffee/" ``` The response body is `Request Rejected`. Under PLM, the Ingress-only App Protect annotations (`appprotect.f5.com/app-protect-policy` and `appprotect.f5.com/app-protect-security-log`) aren't supported. Use the `k8s.nginx.org/v1` Policy resource and the `nginx.com/policies` annotation instead, as shown above. ## Verify bundles on disk Confirm the compiled bundles are present in the ingress controller pod: ```shell NIC_POD=$(kubectl get pods --namespace nginx-ingress \ --selector app.kubernetes.io/name=nginx-ingress \ --output jsonpath='{.items[0].metadata.name}') kubectl exec --namespace nginx-ingress $NIC_POD --container nginx-ingress -- \ ls -ltr /etc/app_protect/bundles/ ``` Expected output: ```text total 1860 -rw------- 1 nginx nginx 1654 Aug 12 10:06 fetched_default_waf-policy_log_0.tgz -rw------- 1 nginx nginx 1898698 Aug 12 13:32 fetched_default_waf-policy_policy.tgz ``` Check the Policy status: ```shell kubectl describe policy waf-policy ``` A `State: Valid` and `Reason: AddedOrUpdated` status confirms the bundles were fetched successfully. ## Troubleshooting - **Policy status is `Warning` with reason `BundleFetchFailed`.** Run `kubectl describe appolicy --namespace security` and confirm `status.bundle.state` is `ready`. If PLM hasn't compiled the resource yet, the Policy fetch can't proceed. - **NGINX Ingress Controller reports the referenced namespace isn't watched.** If `controller.watchNamespace` is set, include the namespace that holds the `APPolicy` and `APLogConf` resources. If `controller.watchSecretNamespace` is set, include the PLM namespace so the controller can observe storage Secret rotation. -------------------------------------------------------------------------------- title: "Use Helm to Install NGINX Ingress Controller with NGINX Plus" url: https://docs.nginx.com/nginx-ingress-controller/install/helm/plus/index.md -------------------------------------------------------------------------------- This page describes how to use Helm to install F5 NGINX Ingress Controller with NGINX Plus. It explains the requirements for NGINX Ingress Controller, creating secrets, how to obtain and install the Helm chart, and what custom resource definitions (CRDs) are installed during the process. By following these instructions, you will finish with a functional NGINX Ingress Controller instance for your Kubernetes cluster. ## Before you begin - An active NGINX Plus subscription (Purchased or trial) - A [supported Kubernetes version](/nic/technical-specifications.md#supported-kubernetes-versions) - A functional Kubernetes cluster - [Helm 3.19+.](https://helm.sh/docs/intro/install) Throughout this page, you will see placeholder values indicated with angular brackets, such as **\**. Replace them accordingly for your installation. ### Quickstart If you do not require extra guidance to understand the installation process or have customization requirements, here are the steps to install NGINX Ingress Controller. You will require your NGINX Plus license JWT to follow these steps: if you do not have it, the steps are explained in the [Download your JSON web token](#download-your-json-web-token) section. **Note:** These commands should be run in the same directory as your **license.jwt** file. Create a secret for the NGINX Plus license: ```shell kubectl create secret generic nplus-license --from-file=license.jwt --type=nginx.com/license ``` Create a secret for the F5 registry: ```shell kubectl create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat license.jwt) \ --docker-password=none ``` Use Helm to install NGINX Ingress Controller with NGINX Plus: ```shell helm install oci://ghcr.io/nginx/charts/nginx-ingress \ --version \ --set controller.image.repository=private-registry.nginx.com/nginx-ic/nginx-plus-ingress \ --set controller.image.tag= \ --set controller.nginxplus=true \ --set controller.serviceAccount.imagePullSecretName=regcred \ --set controller.mgmt.licenseTokenSecretName=nplus-license ``` **Note:** The `edge` version **is not intended for production use**. It is intended for testing and development purposes only. If you'd like to test the latest changes in NGINX Ingress Controller before a new release, you can install the `edge` version, which is built from the `main` branch of the [NGINX Ingress Controller repository](https://github.com/nginx/kubernetes-ingress). You can install the `edge` version by specifying the `--version` flag with the value `0.0.0-edge`: ```shell helm install oci://ghcr.io/nginx/charts/nginx-ingress \ --version 0.0.0-edge \ --set controller.image.repository=private-registry.nginx.com/nginx-ic/nginx-plus-ingress \ --set controller.image.tag= \ --set controller.nginxplus=true \ --set controller.serviceAccount.imagePullSecretName=regcred \ --set controller.mgmt.licenseTokenSecretName=nplus-license ``` ## Download your JSON web token 1. Log in to [MyF5](https://my.f5.com/manage/s/). 1. Go to **My Products & Plans > Subscriptions** to see your active subscriptions. 1. Find your NGINX subscription, and select the **Subscription ID** for details. 1. Download the **JSON Web Token** file from the subscription page. ## Create license and registry secrets **Note:** The commands in the rest of this document should be run in the same directory as your **license.jwt** file. JWTs are sensitive information and should be stored securely. Delete them after use to prevent unauthorized access. Once you have obtained your license JWT, create a Kubernetes secret using `kubectl create`: ```shell kubectl create secret generic nplus-license --from-file license.jwt ``` #### Example output ```text secret/nplus-license created ``` Then create another Kubernetes secret to allow interactions with the F5 registry: ```shell kubectl create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat license.jwt) \ --docker-password=none ``` #### Example output ```text secret/regcred created ``` ## Install the Helm chart You have two options for installing the Helm chart: directly from the F5 registry, or using the source. ### F5 Registry To install NGINX Ingress Controller using the F5 registry, run this command with your release name: ``` helm install oci://ghcr.io/nginx/charts/nginx-ingress \ --version \ --set controller.image.repository=private-registry.nginx.com/nginx-ic/nginx-plus-ingress \ --set controller.image.tag= \ --set controller.nginxplus=true \ --set controller.serviceAccount.imagePullSecretName=regcred \ --set controller.mgmt.licenseTokenSecretName=nplus-license ``` #### Example output ```text Pulled: ghcr.io/nginx/charts/nginx-ingress: Digest: sha256:bb452d593c31b6be39f459f9604882e170227429821bac01e7ddd7da16d91ba1 NAME: h4-plus-registry LAST DEPLOYED: Fri Nov 28 14:47:15 2025 NAMESPACE: default STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None NOTES: NGINX Ingress Controller has been installed. For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/ Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ ``` ### From source To install NGINX Ingress Controller from source, first pull the chart by running this command: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress --untar --version ``` #### Example output ```text Pulled: ghcr.io/nginx/charts/nginx-ingress: Digest: sha256:bb452d593c31b6be39f459f9604882e170227429821bac01e7ddd7da16d91ba1 ``` Then use the `cd` command to change your working directory to _nginx-ingress_: ```shell cd nginx-ingress ``` Finally, install the chart with your release name with `helm install`: ```shell helm install . --set controller.image.repository=private-registry.nginx.com/nginx-ic/nginx-plus-ingress --set controller.nginxplus=true ``` #### Example output ```text NAME: h4-plus-source LAST DEPLOYED: Fri Nov 28 14:49:14 2025 NAMESPACE: default STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None NOTES: NGINX Ingress Controller has been installed. For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/ Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ ``` ## Verify the deployment To verify that NGINX Ingress Controller has been installed correctly, you can review `ingressclasses` with `kubectl get`: ```shell kubectl get ingressclasses ``` #### Example output ```text NAME CONTROLLER PARAMETERS AGE nginx nginx.org/ingress-controller 33s ``` ## Custom Resource Definitions When installing the chart, Helm will install the required CRDs. Without them, NGINX Ingress Controller pods will not become _Ready_. If you do not use the custom resources that require those CRDs, add the parameter `--skip-crds` in your `helm install` command. The following chart parameters should be set to `false`: - `controller.enableCustomResources` - `controller.appprotect.enable` - `controller.appprotectdos.enable` ## Next steps - [NGINX Ingress Controller Helm chart parameters](/nic/install/helm/parameters.md) - [Security recommendations](/nic/configuration/security.md) - [Basic configuration](/nic/configuration/ingress-resources/basic-configuration.md) - [Extensibility with NGINX Plus](/nic/overview/nginx-plus.md) -------------------------------------------------------------------------------- title: "Use Helm to Install NGINX Ingress Controller LTS with NGINX Plus" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/helm/plus/index.md -------------------------------------------------------------------------------- This page describes how to use Helm to install F5 NGINX Ingress Controller LTS with NGINX Plus. It explains the requirements for NGINX Ingress Controller LTS, creating secrets, how to obtain and install the Helm chart, and what custom resource definitions (CRDs) are installed during the process. By following these instructions, you will finish with a functional NGINX Ingress Controller LTS instance for your Kubernetes cluster. ## Before you begin - An active NGINX Plus subscription (Purchased or trial) - A [supported Kubernetes version](/nic/lts/technical-specifications.md#supported-kubernetes-versions) - A functional Kubernetes cluster - [Helm 3.19+.](https://helm.sh/docs/intro/install) Throughout this page, you will see placeholder values indicated with angular brackets, such as **\**. Replace them accordingly for your installation. ### Quickstart If you do not require extra guidance to understand the installation process or have customization requirements, here are the steps to install NGINX Ingress Controller LTS. You will require your NGINX Plus license JWT to follow these steps: if you do not have it, the steps are explained in the [Download your JSON web token](#download-your-json-web-token) section. **note:** These commands should be run in the same directory as your **license.jwt** file. Create a secret for the NGINX Plus license: ```shell kubectl create secret generic nplus-license --from-file license.jwt ``` Create a secret for the F5 registry: ```shell kubectl create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat license.jwt) \ --docker-password=none ``` Use Helm to install NGINX Ingress Controller LTS with NGINX Plus: ```shell helm install oci://ghcr.io/nginx/charts/nginx-ingress-lts \ --version \ --set controller.image.repository=private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress \ --set controller.image.tag= \ --set controller.nginxplus=true \ --set controller.serviceAccount.imagePullSecretName=regcred \ --set controller.mgmt.licenseTokenSecretName=nplus-license ``` ## Download your JSON web token 1. Log in to [MyF5](https://my.f5.com/manage/s/). 1. Go to **My Products & Plans > Subscriptions** to see your active subscriptions. 1. Find your NGINX subscription, and select the **Subscription ID** for details. 1. Download the **JSON Web Token** file from the subscription page. ## Create license and registry secrets **note:** The commands in the rest of this document should be run in the same directory as your **license.jwt** file. JWTs are sensitive information and should be stored securely. Delete them after use to prevent unauthorized access. Once you have obtained your license JWT, create a Kubernetes secret using `kubectl create`: ```shell kubectl create secret generic nplus-license --from-file license.jwt ``` #### Example output ```text secret/nplus-license created ``` Then create another Kubernetes secret to allow interactions with the F5 registry: ```shell kubectl create secret docker-registry regcred \ --docker-server=private-registry.nginx.com \ --docker-username=$(cat license.jwt) \ --docker-password=none ``` #### Example output ```text secret/regcred created ``` ## Install the Helm chart You have two options for installing the Helm chart: directly from the F5 registry, or using the source. ### F5 Registry To install NGINX Ingress Controller LTS using the F5 registry, run this command with your release name: ``` helm install oci://ghcr.io/nginx/charts/nginx-ingress-lts \ --version \ --set controller.image.repository=private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress \ --set controller.image.tag= \ --set controller.nginxplus=true \ --set controller.serviceAccount.imagePullSecretName=regcred \ --set controller.mgmt.licenseTokenSecretName=nplus-license ``` #### Example output ```text Pulled: ghcr.io/nginx/charts/nginx-ingress: Digest: sha256:bb452d593c31b6be39f459f9604882e170227429821bac01e7ddd7da16d91ba1 NAME: h4-plus-registry LAST DEPLOYED: Fri Nov 28 14:47:15 2025 NAMESPACE: default STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None NOTES: NGINX Ingress Controller LTS has been installed. For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/ Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ ``` ### From source To install NGINX Ingress Controller LTS from source, first pull the chart by running this command: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress-lts --untar --version ``` #### Example output ```text Pulled: ghcr.io/nginx/charts/nginx-ingress-lts: Digest: sha256:bb452d593c31b6be39f459f9604882e170227429821bac01e7ddd7da16d91ba1 ``` Then use the `cd` command to change your working directory to _nginx-ingress_: ```shell cd nginx-ingress ``` Finally, install the chart with your release name with `helm install`: ```shell helm install . --set controller.image.repository=private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress --set controller.nginxplus=true ``` #### Example output ```text NAME: h4-plus-source LAST DEPLOYED: Fri Nov 28 14:49:14 2025 NAMESPACE: default STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None NOTES: NGINX Ingress Controller LTS has been installed. For release notes for this version please see: https://docs.nginx.com/nginx-ingress-controller/releases/ Installation and upgrade instructions: https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/ ``` ## Verify the deployment To verify that NGINX Ingress Controller LTS has been installed correctly, you can review `ingressclasses` with `kubectl get`: ```shell kubectl get ingressclasses ``` #### Example output ```text NAME CONTROLLER PARAMETERS AGE nginx nginx.org/ingress-controller 33s ``` ## Custom Resource Definitions When installing the chart, Helm will install the required CRDs. Without them, NGINX Ingress Controller LTS pods will not become _Ready_. If you do not use the custom resources that require those CRDs, add the parameter `--skip-crds` in your `helm install` command. The following chart parameters should be set to `false`: - `controller.enableCustomResources` ## Next steps - [NGINX Ingress Controller LTS Helm chart parameters](/nic/lts/install/helm/parameters.md) - [Security recommendations](/nic/lts/configuration/security.md) - [Basic configuration](/nic/lts/configuration/ingress-resources/basic-configuration.md) - [Extensibility with NGINX Plus](/nic/lts/overview/nginx-plus.md) -------------------------------------------------------------------------------- title: "Policy specification" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/policy-specification/index.md -------------------------------------------------------------------------------- A `Policy` resource defines exactly one policy type under `.spec`. Supported policy types are: - `accessControl` - `rateLimit` - `apiKey` - `basicAuth` - `jwt` - `ingressMTLS` - `egressMTLS` - `oidc` - `oidcNative` - `cache` - `cors` - `waf` - `externalAuth` - `hsts` | Policy type | Description | VirtualServer / VirtualServerRoute | Ingress | | --- | --- | --- | --- | | [`accessControl`](/nic/configuration/policy-resource/policy-reference.md#accesscontrol) | Allows or denies requests based on the client IP address. | Yes | Yes, with `nginx.org/policies` | | [`cors`](/nic/configuration/policy-resource/policy-reference.md#cors) | Configures Cross-Origin Resource Sharing (CORS) headers. | Yes | Yes, with `nginx.org/policies` | | [`egressMTLS`](/nic/configuration/policy-resource/policy-reference.md#egressmtls) | Configures mutual TLS (mTLS) authentication and certificate verification for upstream connections. | Yes | Yes, with `nginx.org/policies` | | [`ingressMTLS`](/nic/configuration/policy-resource/policy-reference.md#ingressmtls) | Configures mTLS client certificate verification. | Yes | Yes, with `nginx.org/policies` | | [`waf`](/nic/configuration/policy-resource/policy-reference.md#waf) | Configures WAF and log configuration policies for [NGINX AppProtect](/nic/integrations/app-protect-waf/configuration.md). | Yes | Yes, with `nginx.com/policies` | | [`externalAuth`](/nic/configuration/policy-resource/policy-reference.md#externalauth) | Authenticates client requests using an external authentication server. | Yes | Yes, with `nginx.org/policies` | | [`rateLimit`](/nic/configuration/policy-resource/policy-reference.md#ratelimit) | Controls the request-processing rate for a defined key. | Yes | No | | [`apiKey`](/nic/configuration/policy-resource/policy-reference.md#apikey) | Authorizes requests that include a valid API key in a specified header or query parameter. | Yes | No | | [`basicAuth`](/nic/configuration/policy-resource/policy-reference.md#basicauth) | Authenticates client requests using HTTP Basic authentication credentials. | Yes | No | | [`jwt`](/nic/configuration/policy-resource/policy-reference.md#jwt-using-a-local-kubernetes-secret) | Authenticates client requests using JSON Web Tokens (JWT). Requires NGINX Plus. | Yes | No | | [`oidc`](/nic/configuration/policy-resource/policy-reference.md#oidc) | Configures NGINX Plus as a relying party for OpenID Connect (OIDC) authentication. | Yes | No | | [`oidcNative`](/nic/configuration/policy-resource/policy-reference.md#oidcnative) | Configures NGINX Plus as a relying party for OIDC authentication using the built-in native module. | Yes | Yes, with `nginx.com/policies` | | [`cache`](/nic/configuration/policy-resource/policy-reference.md#cache) | Configures proxy caching for serving cached content. | Yes | No | | [`hsts`](/nic/configuration/policy-resource/policy-reference.md#hsts) | Configures [HTTP Strict Transport Security](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/) (HSTS) to enforce secure connections to the server. | Yes | No | **note:** NGINX Ingress Controller added Policy resource support for Ingress objects through the [`nginx.org/policies`](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md) annotation in v5.4.0. ## Important rule: one policy type per resource A `Policy` resource must define exactly one policy type under `.spec`. If you need multiple behaviors, create multiple policies and reference them together. The following example is valid: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: allow-localhost spec: accessControl: allow: - 10.0.0.0/8 ``` The following example is **not** valid, because it defines two policy types in the same resource: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: invalid-policy spec: accessControl: allow: - 10.0.0.0/8 cors: allowOrigin: - https://example.com ``` ## What's next Learn how to [apply policies to resources](/nic/configuration/policy-resource/applying-policies.md). -------------------------------------------------------------------------------- title: "Policy specification" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/policy-resource/policy-specification/index.md -------------------------------------------------------------------------------- Below is an example of a policy that allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: allow-localhost spec: accessControl: allow: - 10.0.0.0/8 ``` |Field | Description | Type | Supported in VS/VSR | Supported in Ingress | | ---| ---| ---| --- | --- | |``accessControl`` | Allows or denies requests based on the client IP address. | [accessControl](#accesscontrol) | Yes | Yes | |``rateLimit`` | Controls the request-processing rate for a defined key. | [rateLimit](#ratelimit) | Yes | No | |``apiKey`` | Authorizes requests that include a valid API key in a specified header or query parameter. | [apiKey](#apikey) | Yes | No | |``basicAuth`` | Authenticates client requests using HTTP Basic authentication credentials. | [basicAuth](#basicauth) | Yes | No | |``jwt`` | Authenticates client requests using JSON Web Tokens. | [jwt](#jwt-using-a-local-kubernetes-secret) | Yes | No | |``ingressMTLS`` | Configures client certificate verification. | [ingressMTLS](#ingressmtls) | Yes | No | |``egressMTLS`` | Configures upstream authentication and certificate verification. | [egressMTLS](#egressmtls) | Yes | No | |``oidc`` | Configures NGINX Plus as a relying party for OpenID Connect (OIDC) authentication. | [OIDC](#oidc) | Yes | No | |``cache`` | Configures proxy caching for serving cached content. | [cache](#cache) | Yes | No | |``cors`` | Configures Cross-Origin Resource Sharing (CORS) headers. | [cors](#cors) | Yes | Yes | **note:** NGINX Ingress Controller LTS added Policy resource support for Ingress objects through the [`nginx.org/policies`](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations.md) annotation in v5.4.0. A policy must include exactly one policy type. ## AccessControl The access control policy configures NGINX to deny or allow requests from clients with the specified IP addresses or subnets. For example, the following policy allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: ```yaml accessControl: allow: - 10.0.0.0/8 ``` In contrast, the following policy does the opposite. It denies access for clients from `10.0.0.0/8` and allows access for any other clients: ```yaml accessControl: deny: - 10.0.0.0/8 ``` **note:** This feature uses the NGINX [ngx_http_access_module](http://nginx.org/en/docs/http/ngx_http_access_module.html). The NGINX Ingress Controller LTS access control policy supports either allow rules or deny rules, but not both, unlike the module itself. |Field | Description | Type | Required | | ---| ---| ---| --- | |``allow`` | Allows access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | |``deny`` | Denies access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | \* an accessControl must include either `allow` or `deny`. | ### AccessControl merging behavior A VirtualServer or VirtualServerRoute can reference multiple access control policies. For example, this configuration references two policies, each with a configured allow list: ```yaml policies: - name: allow-policy-one - name: allow-policy-two ``` When a resource references more than one access control policy, NGINX Ingress Controller LTS merges the contents into a single allow list or a single deny list. NGINX Ingress Controller LTS doesn't support referencing both allow and deny policies together, as shown in the following example. If a resource references both allow and deny lists, NGINX Ingress Controller LTS uses only the allow list policies. ```yaml policies: - name: deny-policy - name: allow-policy-one - name: allow-policy-two ``` ## RateLimit The rate limit policy configures NGINX to limit the processing rate of requests. For example, the following policy limits all subsequent requests from a single IP address once the rate exceeds 10 requests per second: ```yaml rateLimit: rate: 10r/s zoneSize: 10M key: ${binary_remote_addr} ``` **note:** This feature uses the NGINX [ngx_http_limit_req_module](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html). **note:** When you turn on the [zone sync feature](/nic/lts/configuration/global-configuration/configmap-resource.md#zone-sync) with NGINX Plus, NGINX Ingress Controller LTS synchronizes the rate limiting zone across all replicas in the cluster. This means all replicas know about requests that other replicas in the cluster have already rate limited. |Field | Description | Type | Required | | ---| ---| ---| --- | |``rate`` | The rate of requests permitted. The rate is specified in requests per second (r/s) or requests per minute (r/m). | ``string`` | Yes | |``key`` | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ``${}``. For example: ``${binary_remote_addr}``. Accepted variables are ``$binary_remote_addr``, ``$request_uri``,``$request_method``, ``$url``, ``$http_``, ``$args``, ``$arg_``, ``$cookie_``, ``$jwt_claim_``. | ``string`` | Yes | |``zoneSize`` | Size of the shared memory zone. Only positive values are allowed. Allowed suffixes are ``k`` or ``m``, if none are present ``k`` is assumed. | ``string`` | Yes | |``delay`` | The delay parameter specifies a limit at which excessive requests become delayed. If not set all excessive requests are delayed. | ``int`` | No | |``noDelay`` | Disables the delaying of excessive requests while requests are being limited. Overrides ``delay`` if both are set. | ``bool`` | No | |``burst`` | Excessive requests are delayed until their number exceeds the ``burst`` size, in which case the request is terminated with an error. | ``int`` | No | |``dryRun`` | Turns on dry run mode. In this mode, NGINX Ingress Controller LTS doesn't apply the rate limit, but it accounts for the number of excessive requests as usual in the shared memory zone. | ``bool`` | No | |``logLevel`` | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are ``info``, ``notice``, ``warn`` or ``error``. Default is ``error``. | ``string`` | No | |``rejectCode`` | Sets the status code to return in response to rejected requests. Must fall into the range ``400..599``. Default is ``503``. | ``int`` | No | |``scale`` | Keeps the rate limit constant by dividing the configured rate by the number of NGINX Ingress Controller LTS pods currently serving traffic. This adjustment keeps the rate limit consistent, even as the number of pods fluctuates due to autoscaling. **This doesn't work correctly if requests from a client aren't distributed evenly across all Ingress Controller pods** (for example, with sticky sessions or long-lived TCP connections with many requests). In these cases, [zone sync](/nic/lts/configuration/global-configuration/configmap-resource.md#zone-sync) gives better results. Turning on `zone-sync` suppresses this setting. | ``bool`` | No | |``condition`` | Add a condition to a rate-limit policy. | [ratelimit.condition](#ratelimitcondition) | No | **note:** For each policy referenced in a VirtualServer or its VirtualServerRoutes, NGINX Ingress Controller LTS generates a single rate limiting zone defined by the [`limit_req_zone`](http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive. If two VirtualServer resources reference the same policy, NGINX Ingress Controller LTS generates two different rate limiting zones, one zone per VirtualServer. ### RateLimit merging behavior A VirtualServer or VirtualServerRoute can reference multiple rate limit policies. For example, this configuration references two policies: ```yaml policies: - name: rate-limit-policy-one - name: rate-limit-policy-two ``` When a resource references more than one rate limit policy, NGINX Ingress Controller LTS configures NGINX to use all referenced rate limits. When you define multiple policies, each additional policy inherits the `dryRun`, `logLevel`, and `rejectCode` parameters from the first policy referenced (`rate-limit-policy-one`, in the example above). ## RateLimit.Condition RateLimit.Condition defines a condition for a rate limit policy. For example: ```yaml condition: jwt: claim: user_details.level match: premium default: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``jwt`` | defines a JWT condition to rate limit against. | [ratelimit.condition.jwt](#ratelimitconditionjwt) | No | |``variables`` | defines a Variable condition to rate limit against. | [ratelimit.condition.variables](#ratelimitconditionvariables) | No | |``default`` | sets the rate limit in this policy to be the default if no conditions are met. In a group of policies with the same condition, only one policy can be the default. | ``bool`` | No | **note:** Conditions (`jwt` or `variables`) are optional, but each policy can only have one. If conditions are used and a request doesn't match any of them, NGINX Ingress Controller LTS applies the `default` policy, if one is defined. Otherwise, if no `default` is set, the request isn't rate limited. Combine the rate limit policy with condition with one or more rate limit policies. For example, you can combine multiple rate limit policies that use [RateLimit.Condition.JWT](#ratelimitconditionjwt) to apply different tiers of rate limit based on the value of a JWT claim. For a practical example of tiered rate limiting by the value of a JWT claim, see the example in the [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/rate-limit-tiered-jwt-claim/README.md). ## RateLimit.Condition.JWT RateLimit.Condition.JWT defines a condition for a rate limit by JWT claim. For example, the following condition applies a rate limit policy only to requests with a JWT claim `user_details.level` with a value `premium`: ```yaml jwt: claim: user_details.level match: premium ``` The rate limit policy applies only to requests that contain a JWT with the specified claim and value. For example, the following JWT payload matches the JWT condition: ```json { "user_details": { "level": "premium" }, "sub": "client1" } ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``claim`` | Claim is the JWT claim to be rate limit by. Nested claims should be separated by ".". | ``string`` | Yes | |``match`` | the value of the claim to match against. | ``string`` | Yes | ## RateLimit.Condition.Variables RateLimit.Condition.Variables defines a condition for a rate limit by NGINX variable. The following example defines a condition for a rate limit policy that applies only to requests with the request method with a value `GET`: ```yaml variables: - name: $request_method match: GET ``` **note:** NGINX Ingress Controller LTS currently supports only one variable at a time. |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | the name of the NGINX variable to be rate limit by. | ``string`` | Yes | |``match`` | the value of the NGINX variable to match against. Values prefixed with the `~` character denote the following is a [regular expression](https://nginx.org/en/docs/http/ngx_http_map_module.html#map). | ``string`` | Yes | ## APIKey The API Key auth policy configures NGINX to authorize client requests based on the presence of a valid API Key in a header or query parameter specified in the policy. **note:** This feature uses the NGINX [ngx_http_auth_request_module](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html) and [NGINX JavaScript (NJS)](https://nginx.org/en/docs/njs/). The policy stores API keys securely using SHA-256 hashing. When a client sends an API Key, NJS hashes it and compares it to the hashed API Key in the NGINX configuration. If the hashed keys match, the NJS subrequest issues a 204 No Content response to the `auth_request` directive, indicating successful authorization. If the client doesn't provide an API Key in the specified header or query parameter, NGINX returns a 401 Unauthorized response. If the client presents an invalid key in the expected header or query parameter, NGINX returns a 403 Forbidden response and denies access. You can use the [errorPages](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#errorpage) property on a route to change the default behavior for 401 or 403 errors. The policy requires at least one header or query parameter. The policy below configures NGINX Ingress Controller LTS to require the API Key `password` in the header "my-header". ```yaml apiKey: suppliedIn: header: - "my-header" clientSecret: api-key-secret ``` ```yaml apiVersion: v1 kind: Secret metadata: name: api-key-secret type: nginx.org/apikey data: client1: cGFzc3dvcmQ= # password ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``suppliedIn`` | `header` or `query`. | | Yes | |``suppliedIn.header`` | An array of headers that the API Key may appear in. | ``string[]`` | No | |``suppliedIn.query`` | An array of query params that the API Key may appear in. | ``string[]`` | No | |``clientSecret`` | The name of the Kubernetes secret that stores the API Key(s). It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/apikey``, and the API Key(s) must be stored in a key: val format where each key is a unique clientID and each value is a unique base64 encoded API Key | ``string`` | Yes | **important:** An APIKey policy must include at least one of the `suppliedIn.header` or `suppliedIn.query` parameters. You can also include both. ### APIKey merging behavior A VirtualServer or VirtualServerRoute can be associated with only one API Key policy per route or subroute. You can replace an API Key policy from a higher level with a different policy defined on a more specific route. For example, a VirtualServer can implement different API Key policies at various levels. In the following configuration, the server-wide `api-key-policy-server` applies to `/backend1` for authorization, because that route has no more specific policy. `/backend2` uses `api-key-policy-route`, defined at the route level. ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: virtual-server spec: host: virtual-server.example.com policies: - name: api-key-policy-server upstreams: - name: backend2 service: backend2-svc port: 80 - name: backend1 service: backend1-svc port: 80 routes: - path: /backend1 action: pass: backend1 - path: /backend2 action: pass: backend2 policies: - name: api-key-policy-route ``` ## BasicAuth The basic auth policy configures NGINX to authenticate client requests using the [HTTP Basic authentication scheme](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication). For example, the following policy rejects all requests that don't include a valid username and password combination in the HTTP header `Authentication`: ```yaml basicAuth: secret: htpasswd-secret realm: "My API" ``` **note:** This feature uses the NGINX [ngx_http_auth_basic_module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html). |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of the Kubernetes secret that stores the Htpasswd configuration. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/htpasswd``, and the config must be stored in the secret under the key ``htpasswd``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. | ``string`` | Yes | |``realm`` | The realm for the basic authentication. | ``string`` | No | ### BasicAuth merging behavior A VirtualServer or VirtualServerRoute can reference multiple basic auth policies, but NGINX Ingress Controller LTS applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: basic-auth-policy-one - name: basic-auth-policy-two ``` In this example, NGINX Ingress Controller LTS uses the configuration from the first policy reference, `basic-auth-policy-one`, and ignores `basic-auth-policy-two`. ## JWT using a local Kubernetes secret The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. The following example policy rejects all requests that don't include a valid JWT in the HTTP header `token`: ```yaml jwt: secret: jwk-secret realm: "My API" token: $http_token ``` You can pass the JWT claims and JOSE headers to the upstream servers. For example: ```yaml action: proxy: upstream: webapp requestHeaders: set: - name: user value: ${jwt_claim_user} - name: alg value: ${jwt_header_alg} ``` This example uses the `requestHeaders` of [Action.Proxy](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#actionproxy) to set the values of two headers that NGINX passes to the upstream servers. The value of the `${jwt_claim_user}` variable is the `user` claim of a JWT. For other claims, use `${jwt_claim_name}`, where `name` is the name of the claim. Nested claims and claims that include a period (`.`) aren't supported. Similarly, use `${jwt_header_name}`, where `name` is the name of a header. This example uses the `alg` header. **note:** This feature uses the NGINX Plus [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of the Kubernetes secret that stores the JWK. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/jwk``, and the JWK must be stored in the secret under the key ``jwk``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. | ``string`` | Yes | |``realm`` | The realm of the JWT. | ``string`` | Yes | |``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | ### JWT merging behavior A VirtualServer or VirtualServerRoute can reference multiple JWT policies, but NGINX Ingress Controller LTS applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: jwt-policy-one - name: jwt-policy-two ``` In this example, NGINX Ingress Controller LTS uses the configuration from the first policy reference, `jwt-policy-one`, and ignores `jwt-policy-two`. ## JWT using JWKS from a remote location The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. You can import the JWKS keys for a JWT policy from a URL, such as a remote server or an identity provider, so you don't have to copy and update them on the Ingress Controller pod. The following example policy rejects all requests that don't include a valid JWT in the HTTP header fetched from the identity provider: ```yaml jwt: realm: MyProductAPI token: $http_token jwksURI: keyCache: 1h ``` **note:** This feature uses the NGINX Plus directive [auth_jwt_key_request](http://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request), part of [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). |Field | Description | Type | Required | Default | | ---| ---| ---| --- | --- | |``jwksURI`` | The remote URI where NGINX Ingress Controller LTS sends the request to retrieve the JSON Web Key set.| ``string`` | Yes | -- | |``keyCache`` | Turns on in-memory caching of JWKS (JSON Web Key Sets) obtained from the ``jwksURI`` and sets a valid time for expiration. | ``string`` | Yes | -- | |``realm`` | The realm of the JWT. | ``string`` | Yes | -- | |``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | -- | |``sniEnabled`` | Turns on SNI (Server Name Indication) for the JWT policy. Use this when the remote server requires SNI to serve the correct certificate. | ``bool`` | No | `false` | |``sniName`` | The SNI name to use when connecting to the remote server. If not set, NGINX Ingress Controller LTS uses the hostname from the ``jwksURI``. | ``string`` | No | -- | |``sslVerify`` | Turns on verification of the JWKS server SSL certificate. | ``bool`` | No | `false` | |``sslVerifyDepth`` | Sets the verification depth in the JWKS server certificates chain. | ``int`` | No | `1` | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate for JWKS server verification. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. | ``string`` | No | -- | **note:** NGINX Ingress Controller LTS turns on content caching by default for each JWT policy, with a default time of 12 hours. This improves resiliency by letting NGINX Ingress Controller LTS retrieve the JWKS (JSON Web Key Set) from the cache even after it expires. ### JWT merging behavior This behavior is similar to using a local Kubernetes secret. A VirtualServer or VirtualServerRoute can reference multiple JWT policies, but NGINX Ingress Controller LTS applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: jwt-policy-one - name: jwt-policy-two ``` In this example, NGINX Ingress Controller LTS uses the configuration from the first policy reference, `jwt-policy-one`, and ignores `jwt-policy-two`. ## IngressMTLS The IngressMTLS policy configures client certificate verification. For example, the following policy verifies a client certificate using the CA certificate specified in `ingress-mtls-secret`: ```yaml ingressMTLS: clientCertSecret: ingress-mtls-secret verifyClient: "on" verifyDepth: 1 ``` Below is an example of `ingress-mtls-secret` using the secret type `nginx.org/ca`: ```yaml kind: Secret metadata: name: ingress-mtls-secret apiVersion: v1 type: nginx.org/ca data: ca.crt: ``` A VirtualServer that references an IngressMTLS policy must: - Turn on [TLS termination](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#virtualservertls). - Reference the policy in the VirtualServer [`spec`](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserver-specification). You can't reference an IngressMTLS policy in a [`route`](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroute) or in a VirtualServerRoute [`subroute`](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroutesubroute). If a resource doesn't meet these conditions, NGINX sends the `500` status code to clients. You can pass the client certificate details, including the certificate, to the upstream servers. For example: ```yaml action: proxy: upstream: webapp requestHeaders: set: - name: client-cert-subj-dn value: ${ssl_client_s_dn} # subject DN - name: client-cert value: ${ssl_client_escaped_cert} # client certificate in the PEM format (urlencoded) ``` This example uses the `requestHeaders` of [Action.Proxy](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#actionproxy) to set the values of the two headers that NGINX passes to the upstream servers. See the [list of embedded variables](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#variables) that `ngx_http_ssl_module` supports, which you can use to pass the client certificate details. **note:** This feature uses the NGINX [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html). ### Configure a certificate revocation list The IngressMTLS policy supports configuring a certificate revocation list (CRL) for your policy, in one of two ways. **note:** You can use only one of these configuration options at a time. 1. Add the `ca.crl` field to the `nginx.org/ca` secret type, which accepts a base64 encoded certificate revocation list. Example: ```yaml kind: Secret metadata: name: ingress-mtls-secret apiVersion: v1 type: nginx.org/ca data: ca.crt: ca.crl: ``` 2. Add the `crlFileName` field to your IngressMTLS policy spec with the name of the CRL file. **note:** Use this configuration option only when your CRL is larger than 1 MiB. Otherwise, use the `nginx.org/ca` secret type to manage your CRL. Example: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: ingress-mtls-policy spec: ingressMTLS: clientCertSecret: ingress-mtls-secret crlFileName: webapp.crl verifyClient: "on" verifyDepth: 1 ``` **important:** When you configure a CRL with the `ingressMTLS.crlFileName` field, keep this additional context in mind: - NGINX Ingress Controller LTS expects the CRL, in this case `webapp.crl`, to be in `/etc/nginx/secrets`. Add a volume mount to the NGINX Ingress Controller LTS deployment to add your CRL to `/etc/nginx/secrets`. - When you update the content of your CRL, for example after you revoke a new certificate, NGINX needs to reload to pick up the latest changes. Depending on your environment, this may require you to update the name of your CRL and apply this update to your `ingress-mtls.yaml` policy so NGINX picks up the latest CRL. See the Kubernetes documentation on [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) to find the best implementation for your environment. |Field | Description | Type | Required | | ---| ---| ---| --- | |``clientCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. | ``string`` | Yes | |``verifyClient`` | Verification for the client. Possible values are ``"on"``, ``"off"``, ``"optional"``, ``"optional_no_ca"``. The default is ``"on"``. | ``string`` | No | |``verifyDepth`` | Sets the verification depth in the client certificates chain. The default is ``1``. | ``int`` | No | |``crlFileName`` | The file name of the Certificate Revocation List. NGINX Ingress Controller LTS looks for this file in `/etc/nginx/secrets`. | ``string`` | No | ### IngressMTLS merging behavior A VirtualServer can reference only a single IngressMTLS policy, and NGINX Ingress Controller LTS ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: ingress-mtls-policy-one - name: ingress-mtls-policy-two ``` In this example, NGINX Ingress Controller LTS uses the configuration from the first policy reference, `ingress-mtls-policy-one`, and ignores `ingress-mtls-policy-two`. ## EgressMTLS The EgressMTLS policy configures upstream authentication and certificate verification. For example, the following policy uses `egress-mtls-secret` to authenticate with the upstream application and `egress-trusted-ca-secret` to verify the certificate of the application: ```yaml egressMTLS: tlsSecret: egress-mtls-secret trustedCertSecret: egress-trusted-ca-secret verifyServer: on verifyDepth: 2 ``` **note:** This feature uses the NGINX [ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html). |Field | Description | Type | Required | | ---| ---| ---| --- | |``tlsSecret`` | The name of the Kubernetes secret that stores the TLS certificate and key. It must be in the same namespace as the Policy resource. The secret must be of the type ``kubernetes.io/tls``, the certificate must be stored in the secret under the key ``tls.crt``, and the key must be stored under the key ``tls.key``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. | ``string`` | No | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. | ``string`` | No | |``verifyServer`` | Turns on verification of the upstream HTTPS server certificate. | ``bool`` | No | |``verifyDepth`` | Sets the verification depth in the proxied HTTPS server certificates chain. The default is ``1``. | ``int`` | No | |``sessionReuse`` | Turns on reuse of SSL sessions to the upstreams. The default is ``true``. | ``bool`` | No | |``serverName`` | Turns on passing of the server name through the ``Server Name Indication`` extension. | ``bool`` | No | |``sslName`` | Lets you override the server name used to verify the certificate of the upstream HTTPS server. | ``string`` | No | |``ciphers`` | Specifies the enabled ciphers for requests to an upstream HTTPS server. The default is ``DEFAULT``. | ``string`` | No | |``protocols`` | Specifies the protocols for requests to an upstream HTTPS server. The default is ``TLSv1 TLSv1.1 TLSv1.2``. | ``string`` | No | > Note: NGINX Ingress Controller LTS doesn't validate the value of ``ciphers`` and ``protocols``. As a result, NGINX can fail to reload the configuration. To make sure the configuration for a VirtualServer or VirtualServerRoute that references the policy applied successfully, check its [status](/nic/lts/configuration/global-configuration/reporting-resources-status.md#virtualserver-and-virtualserverroute-resources). Validation for these fields is planned for a future release. | ### EgressMTLS merging behavior A VirtualServer or VirtualServerRoute can reference multiple EgressMTLS policies, but NGINX Ingress Controller LTS applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: egress-mtls-policy-one - name: egress-mtls-policy-two ``` In this example, NGINX Ingress Controller LTS uses the configuration from the first policy reference, `egress-mtls-policy-one`, and ignores `egress-mtls-policy-two`. ## OIDC **tip:** This feature is turned off by default. To turn it on, set the [enable-oidc](/nic/lts/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-oidc) command-line argument of NGINX Ingress Controller LTS. The OIDC policy configures NGINX Plus as a relying party for OpenID Connect authentication. For example, the following policy uses the client ID `nginx-plus` and the client secret `oidc-secret` to authenticate with the OpenID Connect provider `https://idp.example.com`: ```yaml spec: oidc: clientID: nginx-plus clientSecret: oidc-secret authEndpoint: https://idp.example.com/openid-connect/auth tokenEndpoint: https://idp.example.com/openid-connect/token jwksURI: https://idp.example.com/openid-connect/certs endSessionEndpoint: https://idp.example.com/openid-connect/logout postLogoutRedirectURI: / accessTokenEnable: true pkceEnable: false ``` NGINX Plus passes the ID of an authenticated user to the backend in the HTTP header `username`. **note:** This feature uses the [reference implementation](https://github.com/nginxinc/nginx-openid-connect/) of NGINX Plus as a relying party for OpenID Connect authentication. ### Prerequisites To use OIDC, turn on [zone synchronization](/nginx/admin-guide/high-availability/zone_sync.md). If you don't set up zone synchronization, NGINX Plus fails to reload. You also need to configure a resolver, which NGINX Plus uses to resolve the IDP authorization endpoint. You can find an example configuration [in the GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v/examples/custom-resources/oidc#step-7---configure-nginx-plus-zone-synchronization-and-resolver). **warning:** The configuration in the example doesn't turn on TLS, so synchronization between replicas happens in clear text. This can expose tokens. ### Limitations The OIDC policy defines a few internal locations that you can't customize: `/_jwks_uri`, `/_token`, `/_refresh`, `/_id_token_validation`, `/logout`. In addition, `/_codexch` is the default value for the redirect URI, and `/_logout` is the default value for the post logout redirect URI. You can customize both. Specifying one of these locations as a route in the VirtualServer or VirtualServerRoute causes a collision, and NGINX Plus fails to reload. |Field | Description | Type | Required | | ---| ---| ---| --- | |``clientID`` | The client ID provided by your OpenID Connect provider. | ``string`` | Yes | |``clientSecret`` | The name of the Kubernetes secret that stores the client secret provided by your OpenID Connect provider. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/oidc``, and the secret stored under the key ``client-secret``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. If you enable PKCE, don't configure this field. | ``string`` | Yes | |``authEndpoint`` | URL for the authorization endpoint provided by your OpenID Connect provider. | ``string`` | Yes | |``authExtraArgs`` | A list of extra URL arguments to pass to the authorization endpoint provided by your OpenID Connect provider. Arguments must be URL encoded, multiple arguments may be included in the list, for example ``[ arg1=value1, arg2=value2 ]`` | ``string[]`` | No | |``tokenEndpoint`` | URL for the token endpoint provided by your OpenID Connect provider. | ``string`` | Yes | |``endSessionEndpoint`` | URL provided by your OpenID Connect provider to request the end user be logged out. | ``string`` | No | |``jwksURI`` | URL for the JSON Web Key Set (JWK) document provided by your OpenID Connect provider. | ``string`` | Yes | |``scope`` | List of OpenID Connect scopes. The scope ``openid`` always needs to be present and others can be added concatenating them with a ``+`` sign, for example ``openid+profile+email``, ``openid+email+userDefinedScope``. The default is ``openid``. | ``string`` | No | |``redirectURI`` | Lets you override the default redirect URI. The default is ``/_codexch``. | ``string`` | No | |``postLogoutRedirectURI`` | URI to redirect to after the logout has been performed. Requires ``endSessionEndpoint``. The default is ``/_logout``. | ``string`` | No | |``zoneSyncLeeway`` | Specifies the maximum timeout in milliseconds for synchronizing ID/access tokens and shared values between Ingress Controller pods. The default is ``200``. | ``int`` | No | |``accessTokenEnable`` | Option of whether Bearer token is used to authorize NGINX to access protected backend. | ``boolean`` | No | |``pkceEnable`` | Turns on Proof Key for Code Exchange. The OpenID client needs to be in public mode. `clientSecret` is not used in this mode. | ``boolean`` | No | |``sslVerify`` | Use this option to turn on TLS verification when calls are made to the IDP endpoints. | ``boolean`` | No | |``verifyDepth`` | Sets the verification depth in the proxied HTTPS server certificates chain. The default is ``1``. | ``int`` | No | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. Otherwise, NGINX Ingress Controller LTS rejects the secret as invalid. | ``string`` | No | **note:** Only one OIDC policy can be referenced in a VirtualServer and its VirtualServerRoutes. However, you can still apply the same policy to different routes in the VirtualServer and VirtualServerRoutes. ### OIDC merging behavior A VirtualServer or VirtualServerRoute can reference only a single OIDC policy, and NGINX Ingress Controller LTS ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: oidc-policy-one - name: oidc-policy-two ``` In this example, NGINX Ingress Controller LTS uses the configuration from the first policy reference, `oidc-policy-one`, and ignores `oidc-policy-two`. ## Cache The cache policy configures proxy caching, which improves performance by storing and serving cached responses to clients instead of proxying every request to upstream servers. For example, the following policy creates a cache zone named "mycache" with 10 MB of memory allocated, and caches all GET response codes for 30 seconds: ```yaml cache: cacheZoneName: "mycache" cacheZoneSize: "10m" allowedCodes: ["any"] allowedMethods: ["GET"] time: "30s" ``` Here's an example with more specific configuration: ```yaml cache: cacheZoneName: "mycache" cacheZoneSize: "100m" allowedCodes: [200, 301, 302] allowedMethods: ["GET", "POST"] time: "5m" levels: "1:2" overrideUpstreamCache: true inactive: "60m" useTempPath: false maxSize: "10g" minFree: "1g" manager: files: 100 sleep: "50ms" threshold: "200ms" cacheKey: "$scheme$host$request_uri" cacheUseStale: [ "error", "timeout", "updating", "http_500" ] cacheRevalidate: true cacheBackgroundUpdate: true cacheMinUses: 1 lock: enable: true timeout: "5s" age: "30s" conditions: noCache: [ "$cookie_nocache", "$arg_nocache" ] bypass: [ "$http_authorization" ] ``` **note:** This feature uses the NGINX [ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path) `proxy_cache_path` and related directives. |Field | Description | Type | Required | | --- | ---| ---| --- | |``cacheZoneName`` | CacheZoneName defines the name of the cache zone. Must start with a lowercase letter,followed by alphanumeric characters or underscores, and end with an alphanumeric character. Single lowercase letters are also allowed. Examples: "cache", "my_cache", "cache1". | ``string`` | Yes | |``cacheZoneSize`` | CacheZoneSize defines the size of the cache zone. Must be a number followed by a size unit: 'k' for kilobytes, 'm' for megabytes, or 'g' for gigabytes. Examples: "10m", "1g", "512k". | ``string`` | Yes | |``allowedCodes`` | AllowedCodes defines which HTTP response codes should be cached. Accepts either: - The string "any" to cache all response codes (must be the only element) - A list of HTTP status codes as integers (100-599) Examples: ["any"], [200, 301, 404], [200]. Invalid: ["any", 200] (cannot mix "any" with specific codes). | ``[]IntOrString`` | No | |``time`` | The default cache time for responses. Required when allowedCodes is specified. Must be a number followed by a time unit: 's' for seconds, 'm' for minutes, 'h' for hours, 'd' for days. Examples: "30s", "5m", "1h", "2d". | ``string`` | No | |``allowedMethods`` | AllowedMethods defines which HTTP methods should be cached. Only "GET", "HEAD", and "POST" are supported by the NGINX `proxy_cache_methods` directive. GET and HEAD are always cached by default even if not specified. Maximum of 3 items allowed. Examples: ["GET"], ["GET", "HEAD", "POST"]. Invalid methods: PUT, DELETE, PATCH, and so on. | ``[]string`` | No | |``levels`` | Levels defines the cache directory hierarchy levels for storing cached files. Must be in format "X:Y" or "X:Y:Z" where X, Y, Z are either 1 or 2. This controls the number of subdirectory levels and their name lengths. Examples: "1:2", "2:2", "1:2:2". Invalid: "3:1", "1:3", "1:2:3". | ``string`` | No | |``overrideUpstreamCache`` | OverrideUpstreamCache controls whether to override upstream cache headers (using the `proxy_ignore_headers` directive). When true, NGINX ignores cache-related headers from upstream servers like Cache-Control, Expires, and so on. Default: false. | ``bool`` | No | |``cachePurgeAllow`` | CachePurgeAllow defines IP addresses or CIDR blocks allowed to purge cache. Examples: ["192.168.1.100", "10.0.0.0/8", "::1"]. | ``[]string`` | No | |``cacheKey`` | CacheKey defines a key for caching (`proxy_cache_key`). By default, "$scheme$proxy_host$uri". Must not contain command execution patterns: $(, `, ;, &&, || | ``string`` | No | |``cacheUseStale`` | CacheUseStale determines in which cases a stale cached response can be used (`proxy_cache_use_stale`). Valid parameters: error, timeout, invalid_header, updating, http_500, http_502, http_503, http_504, http_403, http_404, http_429, off. | ``[]string`` | No | |``cacheRevalidate`` | CacheRevalidate turns on revalidation of expired cache items using conditional requests (`proxy_cache_revalidate`). Uses "If-Modified-Since" and "If-None-Match" header fields. | ``bool`` | No | |``cacheBackgroundUpdate`` | CacheBackgroundUpdate lets NGINX start a background subrequest to update an expired cache item (`proxy_cache_background_update`). NGINX returns a stale cached response to the client while it updates the cache. | ``bool`` | No | |``cacheMinUses`` | CacheMinUses sets the number of requests after which NGINX Ingress Controller LTS caches the response (`proxy_cache_min_uses`). | ``integer`` | No | |``inactive`` | Inactive sets the time after which cached data that are not accessed get removed from the cache (inactive parameter). By default, inactive is set to 10 minutes. | ``string`` | No | |``maxSize`` | MaxSize sets the maximum cache size (max_size parameter). When the size is exceeded, the cache manager removes the least recently used data. | ``string`` | No | |``minFree`` | MinFree sets the minimum amount of free space required on the file system with cache (min_free parameter). When there is not enough free space, the cache manager removes the least recently used data. | ``string`` | No | |``useTempPath`` | UseTempPath controls whether temporary files and the cache are put on different file systems (use_temp_path parameter). If set to false, NGINX puts temporary files directly in the cache directory (use_temp_path=off). Default: false (use_temp_path=off, which puts temp files directly in the cache directory for better performance). | ``bool`` | No | |``manager`` | Manager configures the cache manager process parameters (manager_files, manager_sleep, manager_threshold). | ``object`` | No | |``manager.files`` | Files sets the maximum number of files that the cache manager deletes in one iteration. During one iteration, the cache manager deletes no more than manager_files items (by default, 100). | ``integer`` | No | |``manager.sleep`` | Sleep sets the pause between cache manager iterations. Between iterations, a pause configured by manager_sleep (by default, 50 milliseconds) is made. | ``string`` | No | |``manager.threshold`` | Threshold sets the maximum duration of one cache manager iteration. The duration of one iteration is limited by manager_threshold (by default, 200 milliseconds). | ``string`` | No | |``lock`` | Lock configures cache locking to prevent multiple identical requests from populating the same cache element simultaneously. | ``object`` | No | |``lock.enable`` | Enable sets whether cache locking is turned on (`proxy_cache_lock`). When on, only one request at a time can populate a new cache element according to the `proxy_cache_key`. | ``bool`` | No | |``lock.timeout`` | Timeout sets a timeout for `proxy_cache_lock`. When the time expires, NGINX passes the request to the proxied server, but it doesn't cache the response. | ``string`` | No | |``lock.age`` | Age sets the maximum time a cache lock can be held (`proxy_cache_lock_age`). If the last request passed to the proxied server for populating a new cache element hasn't completed within the specified time, NGINX may pass one more request. | ``string`` | No | |``conditions`` | Conditions defines when responses should not be cached or taken from cache. | ``object`` | No | |``conditions.noCache`` | NoCache defines conditions under which the response won't be saved to a cache (`proxy_no_cache`). If at least one value of the string parameters isn't empty and isn't equal to "0", NGINX doesn't save the response. | ``[]string`` | No | |``conditions.bypass`` | Bypass defines conditions under which the response won't be taken from a cache (`proxy_cache_bypass`). If at least one value of the string parameters isn't empty and isn't equal to "0", NGINX doesn't take the response from the cache. | ``[]string`` | No | ### Cache merging behavior A VirtualServer or VirtualServerRoute can reference multiple cache policies, but NGINX Ingress Controller LTS applies only the first one. It ignores every subsequent reference. ## CORS The CORS policy configures Cross-Origin Resource Sharing (CORS) headers. **note:** This feature uses the NGINX `add_header` directive. Below is an example of a CORS policy configuring all the available options: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: cors-policy spec: cors: allowOrigin: - "https://test.example.com" - "https://app.example.com" - "https://admin.example.com" allowMethods: - "GET" - "POST" - "PUT" allowHeaders: - "Content-Type" - "Authorization" - "X-Requested-With" - "X-API-Key" allowCredentials: true exposeHeaders: - "X-Total-Count" - "X-Page-Size" - "X-RateLimit-Remaining" - "X-RateLimit-Reset" maxAge: 3600 ``` |Field | Description | Type | Required | | --- | ---| ---| --- | |``allowOrigin`` | AllowOrigin defines the origins that are allowed to make cross-origin requests. Can be exact domains, single wildcards, or `*` for all origins. Examples: ["https://example.com", "https://*.mydomain.com", "*"] Security: When allowCredentials is true, wildcard "*" is not allowed. The server must specify explicit origins for credentialed requests. |``array[string]`` | Yes | |``allowMethods`` | AllowMethods defines the HTTP methods that are allowed for cross-origin requests. | ``array[string]`` | No | |``allowHeaders`` | AllowHeaders defines the headers that are allowed in cross-origin requests. Common safe headers: ["Accept", "Accept-Language", "Content-Language", "Content-Type"] Custom headers: ["Authorization", "X-Requested-With", "X-Custom-Header"] | ``array[string]`` | No | |``allowCredentials`` | AllowCredentials indicates whether the response to the request can be exposed when the credentials flag is true. When used as part of a response to a preflight request, this indicates whether the actual request can be made using credentials. | ``boolean`` | No | |``exposeHeaders`` | ExposeHeaders defines the headers that browsers are allowed to access. Use this field to expose additional custom headers to the browser. Example: ["X-Total-Count", "X-Page-Size", "X-RateLimit-Remaining"] Note: Set-Cookie headers cannot be exposed through CORS per official MDN specification. | ``array[string]`` | No | |``maxAge`` | MaxAge defines how long (in seconds) the results of a preflight request can be cached. Default: 86400 (24 hours). | ``integer`` | No | **note:** If CORS is currently configured in deployments using `snippets` or `responseHeaders.add`, migrate those settings to the CORS policy and remove the duplicate configuration. ### CORS merging behavior A VirtualServer or VirtualServerRoute can reference multiple CORS policies, but NGINX Ingress Controller LTS applies only the first one. It ignores every subsequent reference. ## What's next Learn how to [manage Policy resources with kubectl](/nic/lts/configuration/policy-resource/using-policy.md). -------------------------------------------------------------------------------- title: "Technical specifications" url: https://docs.nginx.com/nginx-ingress-controller/lts/technical-specifications/index.md -------------------------------------------------------------------------------- This page describes technical specifications for F5 NGINX Ingress Controller LTS, such as its version compatibility with Kubernetes and other NGINX software. ## Supported NGINX Ingress Controller LTS versions NGINX Ingress Controller LTS is a feature-frozen release supported for 36 months from the date of release. Security patches and stability bug fixes are delivered as patch revisions throughout the support period. We test NGINX Ingress Controller LTS on a range of Kubernetes platforms for each release, and list them in the [Changelog](/nic/lts/changelog). We provide technical support for NGINX Ingress Controller LTS on any Kubernetes platform that is currently supported by its provider, and that passes the [Kubernetes conformance tests](https://www.cncf.io/certification/software-conformance/). We provide technical support for F5 customers using NGINX Ingress Controller LTS for 36 months from the date of release. NGINX Ingress Controller supports the following versions of [NGINX Plus](/nginx/) and [Kubernetes](https://kubernetes.io/): | NIC LTS version | Kubernetes versions tested | NIC Helm Chart version | NGINX Plus version | End of Technical Support | | --- | --- | --- | --- | --- | | | 1.30 - 1.36 | | R37.0.5.1 | June 4, 2029 | ## Supported Docker images We provide the following Docker images, which include NGINX Plus bundled with the Ingress Controller binary. ### Images with NGINX Plus for LTS 2026 NGINX Ingress Controller LTS image include NGINX Plus LTS R37.0 | Name | Base image |
Additional modules
| F5 Container Registry Image | Architectures | Based on CR | | ---| ---| --- | --- | --- | --- | |Debian-based image | ``debian:13-slim`` | NJS (NGINX JavaScript)
OpenTelemetry
Agent (NGINX Agent 3) | ``private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress: `` | arm64
amd64 | 5.4.3 | -------------------------------------------------------------------------------- title: "Technical specifications" url: https://docs.nginx.com/nginx-ingress-controller/technical-specifications/index.md -------------------------------------------------------------------------------- This page describes technical specifications for F5 NGINX Ingress Controller, such as its version compatibility with Kubernetes and other NGINX software. ## Supported NGINX Ingress Controller versions We recommend using the latest release of NGINX Ingress Controller, and provides software updates for the most recent release. We test NGINX Ingress Controller on a range of Kubernetes platforms for each release, and list them in the [Changelog](/nic/changelog). We provide technical support for NGINX Ingress Controller on any Kubernetes platform that is currently supported by its provider, and that passes the [Kubernetes conformance tests](https://www.cncf.io/certification/software-conformance/). We provide technical support for F5 customers who are using the most recent version of NGINX Ingress Controller, and any version released within two years of the current release. NGINX Ingress Controller supports the following versions of [NGINX Plus](/nginx/) and [Kubernetes](https://kubernetes.io/): | NIC version | Kubernetes versions tested | NIC Helm Chart version | NIC Operator version | NGINX / NGINX Plus version | | --- | --- | --- | --- | --- | | | 1.30 - 1.36 | | | 1.31.4 / R37.1.0.1 | | 5.5.4 | 1.29 - 1.36 | 2.6.4 | 3.6.5 | 1.31.3 / R37.0.3.1 | | 5.4.3 | 1.28 - 1.35 | 2.5.3 | 3.5.4 | 1.31.1 / R37.0.1.1 | | 5.3.4 | 1.27 - 1.35 | 2.4.4 | 3.4.4 | 1.29.5 / R36 P2 | | 5.2.1 | 1.27 - 1.34 | 2.3.1 | 3.3.1 | 1.29.1 / R35 | | 5.1.1 | 1.25 - 1.33 | 2.2.2 | 3.2.3 | 1.29.1 / R35 | | 5.0.0 | 1.25 - 1.32 | 2.1.0 | 3.1.0 | 1.27.4 / R34 | | 4.0.1 | 1.25 - 1.32 | 2.0.1 | 3.0.1 | 1.27.4 / R33 P2 | | 3.7.2 | 1.25 - 1.31 | 1.4.2 | 2.4.2 | 1.27.2 / R32 P1 | ## Supported Docker images We provide the following Docker images, which include NGINX or NGINX Plus bundled with the Ingress Controller binary. ### Images with NGINX _All images include NGINX 1.31.4._ |
Name
|
Base image
| DockerHub image | Architectures | | ---| --- | --- | --- | |Alpine-based image | ``nginx:1.31.4-alpine``,
based on on ``alpine:3.24`` | ``nginx/nginx-ingress:-alpine`` | arm64
amd64 | |Debian-based image | ``nginx:1.31.4``,
based on on ``debian:13-slim`` | ``nginx/nginx-ingress:`` | arm64
amd64 | |Ubi-based image | ``redhat/ubi10-minimal`` | ``nginx/nginx-ingress:-ubi`` | arm64
amd64 | ### Images with NGINX Plus NGINX Plus images include NGINX Plus R37.1.0.1. #### F5 Container registry NGINX Plus images are available through the F5 Container registry `private-registry.nginx.com`, explained in the [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) and [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) topics. | Name | Base image |
Additional modules
| F5 Container Registry Image | Architectures | | ---| ---| --- | --- | --- | |Alpine-based image | ``alpine:3.24`` | NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic/nginx-plus-ingress:-alpine` | arm64
amd64 | |Alpine-based image with FIPS inside | ``alpine:3.24`` | NJS (NGINX JavaScript)
OpenTelemetry
FIPS module and OpenSSL configuration
NGINX Agent 3 | `nginx-ic/nginx-plus-ingress:-alpine-fips` | arm64
amd64 | |Alpine-based image with F5 WAF for NGINX & FIPS inside | ``alpine:3.24`` | F5 WAF for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
FIPS module and OpenSSL configuration
NGINX Agent 2 | `nginx-ic-nap/nginx-plus-ingress:-alpine-fips` | amd64 | |Alpine-based image with F5 WAF for NGINX & FIPS inside and NGINX Agent 3 | ``alpine:3.24`` | F5 WAF for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
FIPS module and OpenSSL configuration
NGINX Agent 3 | `nginx-ic-nap/nginx-plus-ingress:-alpine-fips-agent` | amd64 | |Alpine-based image with F5 WAF for NGINX v5 & FIPS inside | ``alpine:3.24`` | F5 WAF for NGINX v5
NJS (NGINX JavaScript)
OpenTelemetry
FIPS module and OpenSSL configuration
NGINX Agent 2 | `nginx-ic-nap-v5/nginx-plus-ingress:-alpine-fips` | amd64 | |Alpine-based image with F5 WAF for NGINX v5 & FIPS inside and NGINX Agent 3 | ``alpine:3.24`` | F5 WAF for NGINX v5
NJS (NGINX JavaScript)
OpenTelemetry
FIPS module and OpenSSL configuration
NGINX Agent 3 | `nginx-ic-nap-v5/nginx-plus-ingress:-alpine-fips-agent` | amd64 | |Debian-based image | ``debian:13-slim`` | NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic/nginx-plus-ingress:` | arm64
amd64 | |Debian-based image with F5 WAF for NGINX | ``debian:13-slim`` | F5 WAF for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 2 | `nginx-ic-nap/nginx-plus-ingress:` | amd64 | |Debian-based image with F5 WAF for NGINX and NGINX Agent 3 | ``debian:13-slim`` | F5 WAF for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-nap/nginx-plus-ingress:-agent` | amd64 | |Debian-based image with F5 WAF for NGINX v5 | ``debian:13-slim`` | F5 WAF for NGINX v5
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 2 | `nginx-ic-nap-v5/nginx-plus-ingress:` | amd64 | |Debian-based image with F5 WAF for NGINX v5 and NGINX Agent 3 | ``debian:13-slim`` | F5 WAF for NGINX v5
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-nap-v5/nginx-plus-ingress:-agent` | amd64 | |Debian-based image with F5 DoS for NGINX | ``debian:13-slim`` | F5 DoS for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-dos/nginx-plus-ingress:` | amd64 | |Debian-based image with F5 WAF for NGINX and F5 DoS for NGINX | ``debian:13-slim`` | F5 WAF for NGINX and F5 DoS for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 2 | `nginx-ic-nap-dos/nginx-plus-ingress:` | amd64 | |Debian-based image with F5 WAF for NGINX, F5 DoS for NGINX and NGINX Agent 3 | ``debian:13-slim`` | F5 WAF for NGINX and F5 DoS for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-nap-dos/nginx-plus-ingress:-agent` | amd64 | |Ubi-based image | ``redhat/ubi10-minimal`` | NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic/nginx-plus-ingress:-ubi` | arm64
amd64 | |Ubi-based image with F5 WAF for NGINX | ``redhat/ubi10-minimal`` | F5 WAF for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 2 | `nginx-ic-nap/nginx-plus-ingress:-ubi` | amd64 | |Ubi-based image with F5 WAF for NGINX and NGINX Agent 3 | ``redhat/ubi10-minimal`` | F5 WAF for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-nap/nginx-plus-ingress:-ubi-agent` | amd64 | |Ubi-based image with F5 WAF for NGINX v5 | ``redhat/ubi10-minimal`` | F5 WAF for NGINX v5
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 2 | `nginx-ic-nap-v5/nginx-plus-ingress:-ubi` | amd64 | |Ubi-based image with F5 WAF for NGINX v5 and NGINX Agent 3 | ``redhat/ubi10-minimal`` | F5 WAF for NGINX v5
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-nap-v5/nginx-plus-ingress:-ubi-agent` | amd64 | |Ubi-based image with F5 DoS for NGINX | ``redhat/ubi10-minimal`` | F5 DoS for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-dos/nginx-plus-ingress:-ubi` | amd64 | |Ubi-based image with F5 WAF for NGINX and F5 DoS for NGINX | ``redhat/ubi10-minimal`` | F5 WAF for NGINX and F5 DoS for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 2 | `nginx-ic-nap-dos/nginx-plus-ingress:-ubi` | amd64 | |Ubi-based image with F5 WAF for NGINX, F5 DoS for NGINX and NGINX Agent 3 | ``redhat/ubi10-minimal`` | F5 WAF for NGINX and F5 DoS for NGINX
NJS (NGINX JavaScript)
OpenTelemetry
NGINX Agent 3 | `nginx-ic-nap-dos/nginx-plus-ingress:-ubi-agent` | amd64 | ### Custom images You can customize an existing Dockerfile or use it as a reference to create a new one, which is necessary when: - Choosing a different base image. - Installing additional NGINX modules. ## Supported Helm versions NGINX Ingress Controller can be [installed](/nic/install/helm.md) using Helm 3.0 or later. ## Supported F5 WAF for NGINX versions **Note:** F5 WAF for NGINX package based installation (previously NGINX App Protect WAF v4) is not supported when `readOnlyRootFilesystem` is enabled. NGINX Ingress Controller supports the following versions of [F5 WAF for NGINX](https://docs.nginx.com/waf/): | NIC Version | NAP-WAF Version | Config Manager | Enforcer | | ------------------- | --------------- | -------------- | -------- | | | 37.1+ | | | | 5.5.4 | 37.0+5.690.0 | 5.14.0 | 5.14.0 | | 5.4.3 | 37.0+5.635.1 | 5.13.1 | 5.13.1 | | 5.3.4 | 35+5.527.0 | 5.11.2 | 5.11.2 | | 5.2.1 | 35+5.527.0 | 5.9.0 | 5.9.0 | | 5.1.1 | 35+5.498 | 5.8.0 | 5.8.0 | | 5.0.0 | 34+5.342 | 5.6.0 | 5.6.0 | | 4.0.1 | 33+5.264 | 5.5.0 | 5.5.0 | | 3.7.2 | 32+5.1 | 5.3.0 | 5.3.0 | | 3.6.2 | 32+5.48 | 5.2.0 | 5.2.0 | -------------------------------------------------------------------------------- title: "The design of NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/overview/design/index.md -------------------------------------------------------------------------------- This document explains how the F5 NGINX Ingress Controller is designed, and how it differs when using NGINX or NGINX Plus. The intended audience for this information is primarily the two following groups: - _Operators_ who want to know how the software works and understand how it can fail. - _Developers_ who want to [contribute](https://github.com/nginx/kubernetes-ingress/blob/main/CONTRIBUTING.md) to the project. We assume that the reader is familiar with core Kubernetes concepts, such as Pods, Deployments, Services, and Endpoints. For an understanding of how NGINX itself works, you can read the ["Inside NGINX: How We Designed for Performance & Scale"](https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/) blog post. For conciseness in diagrams, NGINX Ingress Controller is often labelled "IC" on this page. ## NGINX Ingress Controller at a high level This figure depicts an example of NGINX Ingress Controller exposing two web applications within a Kubernetes cluster to clients on the internet: ![image](/nic/ic-high-level.png) **Note:** For simplicity, necessary Kubernetes resources like Deployments and Services aren't shown, which Admin and the users also need to create. The figure shows: - A _Kubernetes cluster_. - Cluster users _Admin_, _User A_ and _User B_, which use the cluster via the _Kubernetes API_. - _Clients A_ and _Clients B_, which connect to the _Applications A_ and _B_ deployed by the corresponding users. - _NGINX Ingress Controller_, deployed in a pod with the namespace _nginx-ingress_ and configured using the _ConfigMap resource_ _nginx-ingress_. A single pod is depicted; at least two pods are typically deployed for redundancy. _NGINX Ingress Controller_ uses the _Kubernetes API_ to get the latest Ingress resources created in the cluster and then configures _NGINX_ according to those resources. - _Application A_ with two pods deployed in the _namespace A_ by _User A_. To expose the application to its clients (_Clients A_) via the host `a.example.com`, _User A_ creates _Ingress A_. - _Application B_ with one pod deployed in the _namespace B_ by _User B_. To expose the application to its clients (_Clients B_) via the host `b.example.com`, _User B_ creates _VirtualServer B_. - _Public Endpoint_, which fronts the _NGINX Ingress Controller_ pod(s). This is typically a standalone TCP load balancer (Cloud, software, or hardware) or a combination of a load balancer with a NodePort service. _Clients A_ and _B_ connect to their applications via the _Public Endpoint_. The yellow and purple arrows represent connections related to the client traffic, and the black arrows represent access to the Kubernetes API. ## The NGINX Ingress Controller pod The NGINX Ingress Controller pod consists of a single container, which includes the following: - The _NGINX Ingress Controller process_, which configures NGINX according to Ingress and other resources created in the cluster. - The _NGINX master process_, which controls NGINX worker processes. - _NGINX worker processes_, which handle the client traffic and load balance the traffic to the backend applications. The following is an architectural diagram depicting how those processes interact together and with some external entities: ![image](/nic/ic-pod.png) This table describes each connection, starting with its type: | # | Protocols | Description | | --- | --- | --- | |1|HTTP| _Prometheus_ fetches NGINX Ingress Controller and NGINX metrics with an NGINX Ingress Controller HTTP endpoint (Default `:9113/metrics`). **Note**: *Prometheus* is not required and the endpoint can be turned off. | |2|HTTPS| _NGINX Ingress Controller_ reads the _Kubernetes API_ for the latest versions of the resources in the cluster and writes to the API to update the handled resources' statuses and emit events. |3|HTTP| _Kubelet_ checks the _NGINX Ingress Controller_ readiness probe (Default `:8081/nginx-ready`) to consider the _NGINX Ingress Controller_ pod [ready](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions). |4|File I/O| When _NGINX Ingress Controller_ starts, it reads the _configuration templates_ from the filesystem necessary for configuration generation. The templates are located in the `/` directory of the container and have the `.tmpl` extension |5|File I/O| _NGINX Ingress Controller_ writes logs to *stdout* and *stderr*, which are collected by the container runtime. |6|File I/O| _NGINX Ingress Controller_ generates NGINX *configuration* based on the resources created in the cluster (See [NGINX Ingress Controller is a Kubernetes Controller](#nginx-ingress-controller-is-a-kubernetes-controller)) and writes it on the filesystem in the `/etc/nginx` folder. The configuration files have a `.conf` extension. |7|File I/O| _NGINX Ingress Controller_ writes _TLS certificates_ and _keys_ from any [TLS Secrets](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets) referenced in the Ingress and other resources to the filesystem. |8|HTTP| _NGINX Ingress Controller_ fetches the [NGINX metrics](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status) via the `unix:/var/lib/nginx/nginx-status.sock` UNIX socket and converts it to Prometheus format used in #1. |9|HTTP| To verify a successful configuration reload, _NGINX Ingress Controller_ ensures at least one _NGINX worker_ has the new configuration. To do that, the *IC* checks a particular endpoint via the `unix:/var/lib/nginx/nginx-config-version.sock` UNIX socket. |10|N/A| To start NGINX, NGINX Ingress Controller runs the `nginx` command, which launches the _NGINX master_. |11|Signal| To reload NGINX, the _NGINX Ingress Controller_ runs the `nginx -s reload` command, which validates the configuration and sends the [reload signal](https://nginx.org/en/docs/control.html) to the *NGINX master*. |12|Signal| To shutdown NGINX, the _NGINX Ingress Controller_ executes `nginx -s quit` command, which sends the graceful shutdown signal to the *NGINX master*. |13|File I/O| The _NGINX master_ sends logs to its _stdout_ and _stderr_, which are collected by the container runtime. |14|File I/O| The _NGINX master_ reads the _TLS cert and keys_ referenced in the configuration when it starts or reloads. |15|File I/O| The _NGINX master_ reads _configuration files_ when it starts or during a reload. |16|Signal| The _NGINX master_ controls the [lifecycle of _NGINX workers_](https://nginx.org/en/docs/control.html#reconfiguration) it creates workers with the new configuration and shutdowns workers with the old configuration. |17|File I/O| An _NGINX worker_ writes logs to its _stdout_ and _stderr_, which are collected by the container runtime. |18|UDP| An _NGINX worker_ sends the HTTP upstream server response latency logs via the Syslog protocol over the UNIX socket `/var/lib/nginx/nginx-syslog.sock` to _NGINX Ingress Controller_. In turn, _NGINX Ingress Controller_ analyzes and transforms the logs into Prometheus metrics. |19|HTTP,HTTPS,TCP,UDP| A _client_ sends traffic to and receives traffic from any of the _NGINX workers_ on ports 80 and 443 and any additional ports exposed by the [GlobalConfiguration resource](/nic//configuration/global-configuration/globalconfiguration-resource.md). |20|HTTP,HTTPS,TCP,UDP| An _NGINX worker_ sends traffic to and receives traffic from the _backends_. |21|HTTP| _Admin_ can connect to the [NGINX stub_status](http://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status) using port 8080 via an _NGINX worker_. By default, NGINX only allows connections from `localhost`. ### Differences with NGINX Plus The previous diagram depicts NGINX Ingress Controller using NGINX. NGINX Ingress Controller with NGINX Plus has the following differences: - To configure NGINX Plus, NGINX Ingress Controller uses [configuration reloads](#reloading-nginx) and the [NGINX Plus API](http://nginx.org/en/docs/http/ngx_http_api_module.html#api). This allows NGINX Ingress Controller to dynamically change the upstream servers. - Instead of the stub status metrics, the extended metrics available from the NGINX Plus API are used. - In addition to TLS certs and keys, NGINX Ingress Controller writes JWKs from the secrets of the type `nginx.org/jwk`, and NGINX workers read them. ## The NGINX Ingress Controller process This section covers the architecture of the NGINX Ingress Controller process, including: - How NGINX Ingress Controller processes a new Ingress resource created by a user. - A summary of how NGINX Ingress Controller works in relation to others Kubernetes Controllers. - The different components of the IC process. ### Processing a new Ingress resource The following diagram depicts how NGINX Ingress Controller processes a new Ingress resource. The the NGINX master and worker processes are represented as a single rectangle, _NGINX_ for simplicity. VirtualServer and VirtualServerRoute resources are indicated similarly. ![image](/nic/ic-process.png) Processing a new Ingress resource involves the following steps: each step corresponds to the arrow on the diagram with the same number: 1. _User_ creates a new Ingress resource. 1. The NGINX Ingress Controller process has a _Cache_ of the resources in the cluster. The _Cache_ includes only the resources NGINX Ingress Controller is concerned with such as Ingresses. The _Cache_ stays in sync with the Kubernetes API by [watching for changes to the resources](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes). 1. Once the _Cache_ has the new Ingress resource, it notifies the _Control Loop_ about the changed resource. 1. The _Control Loop_ gets the latest version of the Ingress resource from the _Cache_. Since the Ingress resource references other resources, such as TLS Secrets, the _Control loop_ gets the latest versions of those referenced resources as well. 1. The _Control Loop_ generates TLS certificates and keys from the TLS Secrets and writes them to the filesystem. 1. The _Control Loop_ generates and writes the NGINX _configuration files_, which correspond to the Ingress resource, and writes them to the filesystem. 1. The _Control Loop_ reloads _NGINX_ and waits for _NGINX_ to successfully reload. As part of the reload: 1. _NGINX_ reads the _TLS certs and keys_. 1. _NGINX_ reads the _configuration files_. 1. The _Control Loop_ emits an event for the Ingress resource and updates its status. If the reload fails, the event includes the error message. ### NGINX Ingress Controller is a Kubernetes controller With the context from the previous sections, we can generalize how NGINX Ingress Controller works: *NGINX Ingress Controller constantly processes both new resources and changes to the existing resources in the cluster. As a result, the NGINX configuration stays up-to-date with the resources in the cluster.* NGINX Ingress Controller is an example of a [Kubernetes Controller](https://kubernetes.io/docs/concepts/architecture/controller/): NGINX Ingress Controller runs a control loop that ensures NGINX is configured according to the desired state (Ingresses and other resources). The desired state is based on the following built-in Kubernetes resources and Custom Resources (CRs): - Layer 7 Load balancing configuration: - Ingresses - VirtualServers (CR) - VirtualServerRoutes (CR) - Layer 7 policies: - Policies (CR) - Layer 4 load balancing configuration: - TransportServers (CR) - Service discovery: - Services - Endpoints - Pods - Secret configuration: - Secrets - Global Configuration: - ConfigMap (only one resource) - GlobalConfiguration (CR, only one resource) NGINX Ingress Controller can watch additional Custom Resources, which are less common and not enabled by default: - [F5 WAF for NGINX resources](/nic/integrations/app-protect-dos/configuration) (APPolicies, APLogConfs, APUserSigs) - IngressLink resource (only one resource) ## NGINX Ingress Controller process components In this section, we describe the components of the NGINX Ingress Controller process and how they interact, including: 1. How NGINX Ingress Controller watches for resources changes. 1. The main components of the NGINX Ingress Controller _Control Loop_. 1. How those components process a resource change. 1. Additional components that are crucial for processing changes. NGINX Ingress Controller is written in [Go](https://golang.org/) and relies heavily on the [Go client for Kubernetes](https://github.com/kubernetes/client-go). Where relevant, we include links to the source code on GitHub. ### Resource caches In an earlier section, [Processing a New Ingress Resource](#processing-a-new-ingress-resource), we mentioned that NGINX Ingress Controller has a cache of the resources in the cluster that stays in sync with the Kubernetes API by watching them for changes. We also mentioned that once the cache is updated, it notifies the control loop about the changed resources. The cache is actually a collection of *informers*. The following diagram shows how changes to resources are processed by NGINX Ingress Controller. ![image](/nic/ic-process-components.png) - For every resource type that NGINX Ingress Controller monitors, it creates an [_Informer_](https://pkg.go.dev/k8s.io/client-go@v0.21.0/tools/cache#SharedInformer). The _Informer_ includes a _Store_ that holds the resources of that type. To keep the _Store_ in sync with the latest versions of the resources in the cluster, the _Informer_ calls the Watch and List _Kubernetes APIs_ for that resource type (see the arrow _1. Watch and List_ on the diagram). - When a change happens in the cluster (for example, a new resource is created), the _Informer_ updates its _Store_ and invokes [_Handlers_](https://pkg.go.dev/k8s.io/client-go@v0.21.0/tools/cache#ResourceEventHandler) (See the arrow _2. Invoke_) for that _Informer_. - NGINX Ingress Controller registers _Handlers_ for every _Informer_. Most of the time, a _Handler_ creates an entry for the affected resource in the _Workqueue_ where a workqueue element includes the type of the resource and its namespace and name (See the arrow _3. Put_). - The _Workqueue_ always tries to drain itself: if there is an element at the front, the queue will remove the element and send it to the _Controller_ by calling a callback function (See the arrow _4. Send_). - The _Controller_ is the primary component of NGINX Ingress Controller, which represents the _Control Loop_, explained in [The Control Loop](#the-control-loop) section. To process a workqueue element, the _Controller_ component gets the latest version of the resource from the _Store_ (See the arrow _5. Get_), reconfigures _NGINX_ according to the resource (See the arrow _6. Reconfigure*_, updates the resource status, and emits an event via the _Kubernetes API_ (See the arrow _7. Update status and emit event_). ### The control loop This section discusses the main components of NGINX Ingress Controller, which comprise the control loop: - [Controller](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/controller.go#L90) - Runs the NGINX Ingress Controller control loop. - Instantiates _Informers_, _Handlers_, the _Workqueue_ and additional helper components. - Includes the sync method), which is called by the _Workqueue_ to process a changed resource. - Passes changed resources to _Configurator_ to re-configure NGINX. - [Configurator](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/configs/configurator.go#L95) - Generates NGINX configuration files, TLS and cert keys, and JWKs based on the Kubernetes resource. - Uses _Manager_ to write the generated files and reload NGINX. - [Manager](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/nginx/manager.go#L52) - Controls the lifecycle of NGINX (starting, reloading, quitting). See [Reloading NGINX](#reloading-nginx) for more details about reloading. - Manages the configuration files, TLS keys and certs, and JWKs. The following diagram shows how the three components interact: ![image](/nic/control-loop.png) #### The Controller sync method The Controller [sync](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/controller.go#L663) method is called by the _Workqueue_ to process a change of a resource. The method determines the _kind_ of the resource and calls the appropriate _sync_ method (Such as _syncIngress_ for Ingress resources). To explain how the sync methods work, we will examine the most important one: the _syncIngress_ method, and describe how it processes a new Ingress resource. ![image](/nic/controller-sync.png) 1. The _Workqueue_ calls the _sync_ method and passes a workqueue element to it that includes the changed resource _kind_ and _key_ (The key is the resource namespace/name such as “default/cafe-ingress”). 1. Using the _kind_, the _sync_ method calls the appropriate sync method and passes the resource key. For Ingress resources, the method is _syncIngress_. 1. _syncIngress_ gets the Ingress resource from the *Ingress Store* using the key. The _Store_ is controlled by the _Ingress Informer_. In the code, we use the helper _storeToIngressLister_ type that wraps the _Store_. 1. _syncIngress_ calls _AddOrUpdateIngress_ of the _Configuration_, passing the Ingress along. The [Configuration](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/configuration.go#L320) is a component that represents a valid collection of load balancing configuration resources (Ingresses, VirtualServers, VirtualServerRoutes, TransportServers), ready to be converted to the NGINX configuration (see the [Configuration section](#configuration) for more details). _AddOrUpdateIngress_ returns a list of [ResourceChanges](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/configuration.go#L59), which must be reflected in the NGINX config. Typically, for a new Ingress resource, the _Configuration_ returns only a single _ResourceChange_. 1. _syncIngress_ calls _processChanges_, which processes the single Ingress _ResourceChange_. 1. _processChanges_ creates an extended Ingress resource (_IngressEx_) that includes the original Ingress resource and its dependencies, such as Endpoints and Secrets, to generate the NGINX configuration. For simplicity, we don’t show this step on the diagram. 1. _processChanges_ calls _AddOrUpdateIngress_ of the _Configurator_ and passes the extended Ingress resource. 1. _Configurator_ generates an NGINX configuration file based on the extended Ingress resource, then: 1. Calls _Manager’s CreateConfig()_ to update the config for the Ingress resource. 1. Calls _Manager’s Reload()_ to reload NGINX. 1. The reload status is propagated from _Manager_ to _processChanges_, and is either a success or a failure with an error message. 1. _processChanges_ calls _updateRegularIngressStatusAndEvent_ to update the status of the Ingress resource and emit an event with the status of the reload: both make an API call to the Kubernetes API. **Additional notes**: - Many details are not included for conciseness: the source code provides the most granular detail. - The _syncVirtualServer_, _syncVirtualServerRoute_, and _syncTransportServer_ methods are similar to _syncIngress_, while other sync methods are different. However, those methods typically find the affected Ingress, VirtualServer, and TransportServer resources and regenerate the configuration for them. - The _Workqueue_ has only a single worker thread that calls the sync method synchronously, meaning the _Control Loop_ processes only one change at a time. #### Helper components There are two additional helper components crucial for processing changes: _Configuration_ and _LocalSecretStore_. ##### Configuration [_Configuration_](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/configuration.go#L320) holds the latest valid state of the NGINX Ingress Controller load balancing configuration resources: Ingresses, VirtualServers, VirtualServerRoutes, TransportServers, and GlobalConfiguration. The _Configuration_ supports add, update and delete operations on the resources. When you invoke these operations on a resource in the Configuration, it performs the following: 1. Validates the object (For add or update) 1. Calculates the changes to the affected resources that are necessary to propagate to the NGINX configuration, returning the changes to the caller. For example, when you add a new Ingress resource, the _Configuration_ returns a change requiring NGINX Ingress Controller to add the configuration for that Ingress to the NGINX configuration files. If you made an existing Ingress resource invalid, the _Configuration_ returns a change requiring NGINX Ingress Controller to remove the configuration for that Ingress from the NGINX configuration files. Additionally, the _Configuration_ ensures that only one Ingress/VirtualServer/TransportServer (TLS Passthrough) resource holds a particular host (For example, cafe.example.com) and only one TransportServer (TCP/UDP) holds a particular listener (Such as port 53 for UDP). This ensures that no host or listener collisions happen in the NGINX configuration. Ultimately, NGINX Ingress Controller ensures the NGINX config on the filesystem reflects the state of the objects in the _Configuration_ at any point in time. ##### LocalSecretStore [_LocalSecretStore_](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/secrets/store.go#L32) (of the _SecretStore_ interface) holds the valid Secret resources and keeps the corresponding files on the filesystem in sync with them. Secrets are used to hold TLS certificates and keys (type `kubernetes.io/tls`), CAs (`nginx.org/ca`), JWKs (`nginx.org/jwk`), and client secrets for an OIDC provider (`nginx.org/oidc`). When _Controller_ processes a change to a configuration resource like Ingress, it creates an extended version of a resource that includes the dependencies (Such as Secrets) necessary to generate the NGINX configuration. _LocalSecretStore_ allows _Controller_ to reference the filesystem for a secret using the secret key (namespace/name). ## Reloading NGINX The following sections describe how NGINX reloads and how NGINX Ingress Controller specifically affects this process. ### How NGINX reloads work Reloading NGINX is necessary to apply new configuration changes and occurs with these steps: 1. The administrator sends a HUP (hangup) signal to the NGINX master process to trigger a reload. 1. The master process brings down the worker processes with the old configuration and starts worker processes with the new configuration. 1. The administrator verifies the reload has successfully finished. The [NGINX documentation](https://nginx.org/en/docs/control.html#reconfiguration) has more details about reloading. #### How to reload NGINX and confirm success The NGINX binary (`nginx`) supports the reload operation with the `-s reload` option. When you run this option: 1. It validates the new NGINX configuration and exits if it is invalid printing the error messages to the stderr. 1. It sends a HUP signal to the NGINX master process and exits. As an alternative, you can send a HUP signal to the NGINX master process directly. Once the reload operation has been invoked with `nginx -s reload`, there is no wait period for NGINX to finish reloading. This means it is the responsibility of an administator to check it is finished, for which there are a few options: - Check if the master process created new worker processes. Two ways are by running `ps` or reading the `/proc` file system. - Send an HTTP request to NGINX, to see if a new worker process responds. This signifies that NGINX reloaded successfully: this method requires additional NGINX configuration, explained below. NGINX reloads take roughly 200ms. The factors affecting reload time are configuration size and details, the number of TLS certificates/keys, enabled modules, and available CPU resources. #### Potential problems Most of the time, if `nginx -s reload` executes, the reload will also succeed. In the rare case a reload fails, the NGINX master process will print the an error message. This is an example: ```shell 2022/07/09 00:56:42 [emerg] 1353#1353: limit_req "one" uses the "$remote_addr" key while previously it used the "$binary_remote_addr" key ``` The operation is graceful; reloading doesn't lead to any traffic loss by NGINX. However, frequent reloads can lead to high memory utilization and potential OOM (Out-Of-Memory) errors, resulting in traffic loss. This can most likely happen if you (1) proxy traffic that utilizes long-lived connections (ex: Websockets, gRPC) and (2) reload frequently. In these scenarios, you can end up with multiple generations of NGINX worker processes that are shutting down which will force old workers to shut down after the timeout). Eventually, all those worker processes can exhaust the system's available memory. Old NGINX workers will not shut down until all connections are terminated either by clients or backends, unless you configure [worker_shutdown_timeout](https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout). Since both the old and new NGINX worker processes coexist during a reload, reloading can lead to two spikes in memory utilization. With a lack of available memory, the NGINX master process can fail to create new worker processes. ### Reloading in NGINX Ingress Controller NGINX Ingress Controller reloads NGINX to apply configuration changes. To facilitate reloading, NGINX Ingress Controller configures a server listening on the Unix socket `unix:/var/lib/nginx/nginx-config-version.sock` that responds with the configuration version for `/configVersion` URI. NGINX Ingress Controller writes the configuration to `/etc/nginx/config-version.conf`. Reloads occur with this sequence of steps: 1. NGINX Ingress Controller updates generated configuration files, including any secrets. 1. NGINX Ingress Controller updates the config version in `/etc/nginx/config-version.conf`. 1. NGINX Ingress Controller runs `nginx -s reload`. If the command fails, NGINX Ingress Controller logs the error and considers the reload failed. 1. If the command succeeds, NGINX Ingress Controller periodically checks for the config version by sending an HTTP request to the config version server on `unix:/var/lib/nginx/nginx-config-version.sock`. 1. Once NGINX Ingress Controller sees the correct config version returned by NGINX, it considers the reload successful. If it doesn't see the correct configuration version after the configurable timeout ( [`-nginx-reload-timeout`](/nic/configuration/global-configuration/command-line-arguments)), NGINX Ingress Controller considers the reload failed. The [NGINX Ingress Controller Control Loop](#the-control-loop) stops during a reload so that it cannot affect configuration files or reload NGINX until the current reload succeeds or fails. ### When NGINX Ingress Controller reloads NGINX NGINX Ingress Controller reloads NGINX every time the Control Loop processes a change that affects the generated NGINX configuration. In general, every time a monitored resource is changed, NGINX Ingress Controller will regenerate the configuration and reload NGINX. There are three special cases: - *Start*. When NGINX Ingress Controller starts, it processes all resources in the cluster and only then reloads NGINX. This avoids a "reload storm" by reloading only once. - *Batch updates*. When NGINX Ingress Controller receives a number of concurrent requests from the Kubernetes API, it will pause NGINX reloads until the task queue is empty. This reduces the number of reloads to minimize the impact of batch updates and reduce the risk of OOM (Out of Memory) errors. - *NGINX Plus*. If NGINX Ingress Controller is using NGINX Plus, it will not reload NGINX Plus for changes to the Endpoints resources. In this case, NGINX Ingress Controller will use the NGINX Plus API to update the corresponding upstreams and skip reloading. -------------------------------------------------------------------------------- title: "The design of NGINX Ingress Controller LTS" url: https://docs.nginx.com/nginx-ingress-controller/lts/overview/design/index.md -------------------------------------------------------------------------------- This document explains how the F5 NGINX Ingress Controller LTS is designed. The intended audience for this information is primarily the two following groups: - _Operators_ who want to know how the software works and understand how it can fail. - _Developers_ who want to [contribute](https://github.com/nginx/kubernetes-ingress/blob/main/CONTRIBUTING.md) to the project. We assume that the reader is familiar with core Kubernetes concepts, such as Pods, Deployments, Services, and Endpoints. For an understanding of how NGINX itself works, you can read the ["Inside NGINX: How We Designed for Performance & Scale"](https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/) blog post. For conciseness in diagrams, NGINX Ingress Controller LTS is often labelled "IC" on this page. ## NGINX Ingress Controller LTS at a high level This figure depicts an example of NGINX Ingress Controller LTS exposing two web applications within a Kubernetes cluster to clients on the internet: ![image](/nic/ic-high-level.png) **note:** For simplicity, necessary Kubernetes resources like Deployments and Services aren't shown, which Admin and the users also need to create. The figure shows: - A _Kubernetes cluster_. - Cluster users _Admin_, _User A_ and _User B_, which use the cluster via the _Kubernetes API_. - _Clients A_ and _Clients B_, which connect to the _Applications A_ and _B_ deployed by the corresponding users. - _NGINX Ingress Controller LTS_, deployed in a pod with the namespace _nginx-ingress_ and configured using the _ConfigMap resource_ _nginx-ingress_. A single pod is depicted; at least two pods are typically deployed for redundancy. _NGINX Ingress Controller LTS_ uses the _Kubernetes API_ to get the latest Ingress resources created in the cluster and then configures _NGINX_ according to those resources. - _Application A_ with two pods deployed in the _namespace A_ by _User A_. To expose the application to its clients (_Clients A_) via the host `a.example.com`, _User A_ creates _Ingress A_. - _Application B_ with one pod deployed in the _namespace B_ by _User B_. To expose the application to its clients (_Clients B_) via the host `b.example.com`, _User B_ creates _VirtualServer B_. - _Public Endpoint_, which fronts the _NGINX Ingress Controller LTS_ pod(s). This is typically a standalone TCP load balancer (Cloud, software, or hardware) or a combination of a load balancer with a NodePort service. _Clients A_ and _B_ connect to their applications via the _Public Endpoint_. The yellow and purple arrows represent connections related to the client traffic, and the black arrows represent access to the Kubernetes API. ## The NGINX Ingress Controller LTS pod The NGINX Ingress Controller LTS pod consists of a single container, which includes the following: - The _NGINX Ingress Controller LTS process_, which configures NGINX according to Ingress and other resources created in the cluster. - The _NGINX master process_, which controls NGINX worker processes. - _NGINX worker processes_, which handle the client traffic and load balance the traffic to the backend applications. The following is an architectural diagram depicting how those processes interact together and with some external entities: ![image](/nic/ic-pod.png) This table describes each connection, starting with its type: | # | Protocols | Description | | --- | --- | --- | |1|HTTP| _Prometheus_ fetches NGINX Ingress Controller LTS and NGINX metrics with an NGINX Ingress Controller LTS HTTP endpoint (Default `:9113/metrics`). **Note**: *Prometheus* is not required and the endpoint can be turned off. | |2|HTTPS| _NGINX Ingress Controller LTS_ reads the _Kubernetes API_ for the latest versions of the resources in the cluster and writes to the API to update the handled resources' statuses and emit events. |3|HTTP| _Kubelet_ checks the _NGINX Ingress Controller LTS_ readiness probe (Default `:8081/nginx-ready`) to consider the _NGINX Ingress Controller LTS_ pod [ready](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions). |4|File I/O| When _NGINX Ingress Controller LTS_ starts, it reads the _configuration templates_ from the filesystem necessary for configuration generation. The templates are located in the `/` directory of the container and have the `.tmpl` extension |5|File I/O| _NGINX Ingress Controller LTS_ writes logs to *stdout* and *stderr*, which are collected by the container runtime. |6|File I/O| _NGINX Ingress Controller LTS_ generates NGINX *configuration* based on the resources created in the cluster (See [NGINX Ingress Controller LTS is a Kubernetes Controller](#nginx-ingress-controller-is-a-kubernetes-controller)) and writes it on the filesystem in the `/etc/nginx` folder. The configuration files have a `.conf` extension. |7|File I/O| _NGINX Ingress Controller LTS_ writes _TLS certificates_ and _keys_ from any [TLS Secrets](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets) referenced in the Ingress and other resources to the filesystem. |8|HTTP| _NGINX Ingress Controller LTS_ fetches the [NGINX metrics](https://nginx.org/en/docs/http/ngx_http_api_module.html) via the NGINX Plus API. |9|HTTP| To verify a successful configuration reload, _NGINX Ingress Controller LTS_ ensures at least one _NGINX worker_ has the new configuration. To do that, the *IC* checks a particular endpoint via the `unix:/var/lib/nginx/nginx-config-version.sock` UNIX socket. |10|N/A| To start NGINX, NGINX Ingress Controller LTS runs the `nginx` command, which launches the _NGINX master_. |11|Signal| To reload NGINX, the _NGINX Ingress Controller LTS_ runs the `nginx -s reload` command, which validates the configuration and sends the [reload signal](https://nginx.org/en/docs/control.html) to the *NGINX master*. |12|Signal| To shutdown NGINX, the _NGINX Ingress Controller LTS_ executes `nginx -s quit` command, which sends the graceful shutdown signal to the *NGINX master*. |13|File I/O| The _NGINX master_ sends logs to its _stdout_ and _stderr_, which are collected by the container runtime. |14|File I/O| The _NGINX master_ reads the _TLS cert and keys_ referenced in the configuration when it starts or reloads. |15|File I/O| The _NGINX master_ reads _configuration files_ when it starts or during a reload. |16|Signal| The _NGINX master_ controls the [lifecycle of _NGINX workers_](https://nginx.org/en/docs/control.html#reconfiguration) it creates workers with the new configuration and shutdowns workers with the old configuration. |17|File I/O| An _NGINX worker_ writes logs to its _stdout_ and _stderr_, which are collected by the container runtime. |18|UDP| An _NGINX worker_ sends the HTTP upstream server response latency logs via the Syslog protocol over the UNIX socket `/var/lib/nginx/nginx-syslog.sock` to _NGINX Ingress Controller LTS_. In turn, _NGINX Ingress Controller LTS_ analyzes and transforms the logs into Prometheus metrics. |19|HTTP,HTTPS,TCP,UDP| A _client_ sends traffic to and receives traffic from any of the _NGINX workers_ on ports 80 and 443 and any additional ports exposed by the [GlobalConfiguration resource](/nic/lts/configuration/global-configuration/globalconfiguration-resource.md). |20|HTTP,HTTPS,TCP,UDP| An _NGINX worker_ sends traffic to and receives traffic from the _backends_. |21|HTTP| _Admin_ can connect to the [NGINX stub_status](http://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status) using port 8080 via an _NGINX worker_. By default, NGINX only allows connections from `localhost`. ### NGINX Plus capabilities NGINX Ingress Controller LTS uses NGINX Plus and provides the following capabilities: - To configure NGINX Plus, NGINX Ingress Controller LTS uses [configuration reloads](#reloading-nginx) and the [NGINX Plus API](http://nginx.org/en/docs/http/ngx_http_api_module.html#api). This allows NGINX Ingress Controller LTS to dynamically change the upstream servers. - Extended metrics available from the NGINX Plus API are used instead of stub status metrics. - In addition to TLS certs and keys, NGINX Ingress Controller LTS writes JWKs from the secrets of the type `nginx.org/jwk`, and NGINX workers read them. ## The NGINX Ingress Controller LTS process This section covers the architecture of the NGINX Ingress Controller LTS process, including: - How NGINX Ingress Controller LTS processes a new Ingress resource created by a user. - A summary of how NGINX Ingress Controller LTS works in relation to others Kubernetes Controllers. - The different components of the IC process. ### Processing a new Ingress resource The following diagram depicts how NGINX Ingress Controller LTS processes a new Ingress resource. The the NGINX master and worker processes are represented as a single rectangle, _NGINX_ for simplicity. VirtualServer and VirtualServerRoute resources are indicated similarly. ![image](/nic/ic-process.png) Processing a new Ingress resource involves the following steps: each step corresponds to the arrow on the diagram with the same number: 1. _User_ creates a new Ingress resource. 1. The NGINX Ingress Controller LTS process has a _Cache_ of the resources in the cluster. The _Cache_ includes only the resources NGINX Ingress Controller LTS is concerned with such as Ingresses. The _Cache_ stays in sync with the Kubernetes API by [watching for changes to the resources](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes). 1. Once the _Cache_ has the new Ingress resource, it notifies the _Control Loop_ about the changed resource. 1. The _Control Loop_ gets the latest version of the Ingress resource from the _Cache_. Since the Ingress resource references other resources, such as TLS Secrets, the _Control loop_ gets the latest versions of those referenced resources as well. 1. The _Control Loop_ generates TLS certificates and keys from the TLS Secrets and writes them to the filesystem. 1. The _Control Loop_ generates and writes the NGINX _configuration files_, which correspond to the Ingress resource, and writes them to the filesystem. 1. The _Control Loop_ reloads _NGINX_ and waits for _NGINX_ to successfully reload. As part of the reload: 1. _NGINX_ reads the _TLS certs and keys_. 1. _NGINX_ reads the _configuration files_. 1. The _Control Loop_ emits an event for the Ingress resource and updates its status. If the reload fails, the event includes the error message. ### NGINX Ingress Controller LTS is a Kubernetes controller With the context from the previous sections, we can generalize how NGINX Ingress Controller LTS works: *NGINX Ingress Controller LTS constantly processes both new resources and changes to the existing resources in the cluster. As a result, the NGINX configuration stays up-to-date with the resources in the cluster.* NGINX Ingress Controller LTS is an example of a [Kubernetes Controller](https://kubernetes.io/docs/concepts/architecture/controller/): NGINX Ingress Controller LTS runs a control loop that ensures NGINX is configured according to the desired state (Ingresses and other resources). The desired state is based on the following built-in Kubernetes resources and Custom Resources (CRs): - Layer 7 Load balancing configuration: - Ingresses - VirtualServers (CR) - VirtualServerRoutes (CR) - Layer 7 policies: - Policies (CR) - Layer 4 load balancing configuration: - TransportServers (CR) - Service discovery: - Services - Endpoints - Pods - Secret configuration: - Secrets - Global Configuration: - ConfigMap (only one resource) - GlobalConfiguration (CR, only one resource) NGINX Ingress Controller LTS can watch additional Custom Resources, which are less common and not enabled by default: - IngressLink resource (only one resource) ## NGINX Ingress Controller LTS process components In this section, we describe the components of the NGINX Ingress Controller LTS process and how they interact, including: 1. How NGINX Ingress Controller LTS watches for resources changes. 1. The main components of the NGINX Ingress Controller LTS _Control Loop_. 1. How those components process a resource change. 1. Additional components that are crucial for processing changes. NGINX Ingress Controller LTS is written in [Go](https://golang.org/) and relies heavily on the [Go client for Kubernetes](https://github.com/kubernetes/client-go). Where relevant, we include links to the source code on GitHub. ### Resource caches In an earlier section, [Processing a New Ingress Resource](#processing-a-new-ingress-resource), we mentioned that NGINX Ingress Controller LTS has a cache of the resources in the cluster that stays in sync with the Kubernetes API by watching them for changes. We also mentioned that once the cache is updated, it notifies the control loop about the changed resources. The cache is actually a collection of *informers*. The following diagram shows how changes to resources are processed by NGINX Ingress Controller LTS. ![image](/nic/ic-process-components.png) - For every resource type that NGINX Ingress Controller LTS monitors, it creates an [_Informer_](https://pkg.go.dev/k8s.io/client-go@v0.21.0/tools/cache#SharedInformer). The _Informer_ includes a _Store_ that holds the resources of that type. To keep the _Store_ in sync with the latest versions of the resources in the cluster, the _Informer_ calls the Watch and List _Kubernetes APIs_ for that resource type (see the arrow _1. Watch and List_ on the diagram). - When a change happens in the cluster (for example, a new resource is created), the _Informer_ updates its _Store_ and invokes [_Handlers_](https://pkg.go.dev/k8s.io/client-go@v0.21.0/tools/cache#ResourceEventHandler) (See the arrow _2. Invoke_) for that _Informer_. - NGINX Ingress Controller LTS registers _Handlers_ for every _Informer_. Most of the time, a _Handler_ creates an entry for the affected resource in the _Workqueue_ where a workqueue element includes the type of the resource and its namespace and name (See the arrow _3. Put_). - The _Workqueue_ always tries to drain itself: if there is an element at the front, the queue will remove the element and send it to the _Controller_ by calling a callback function (See the arrow _4. Send_). - The _Controller_ is the primary component of NGINX Ingress Controller LTS, which represents the _Control Loop_, explained in [The Control Loop](#the-control-loop) section. To process a workqueue element, the _Controller_ component gets the latest version of the resource from the _Store_ (See the arrow _5. Get_), reconfigures _NGINX_ according to the resource (See the arrow _6. Reconfigure*_, updates the resource status, and emits an event via the _Kubernetes API_ (See the arrow _7. Update status and emit event_). ### The control loop This section discusses the main components of NGINX Ingress Controller LTS, which comprise the control loop: - [Controller](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/controller.go#L90) - Runs the NGINX Ingress Controller LTS control loop. - Instantiates _Informers_, _Handlers_, the _Workqueue_ and additional helper components. - Includes the sync method), which is called by the _Workqueue_ to process a changed resource. - Passes changed resources to _Configurator_ to re-configure NGINX. - [Configurator](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/configs/configurator.go#L95) - Generates NGINX configuration files, TLS and cert keys, and JWKs based on the Kubernetes resource. - Uses _Manager_ to write the generated files and reload NGINX. - [Manager](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/nginx/manager.go#L52) - Controls the lifecycle of NGINX (starting, reloading, quitting). See [Reloading NGINX](#reloading-nginx) for more details about reloading. - Manages the configuration files, TLS keys and certs, and JWKs. The following diagram shows how the three components interact: ![image](/nic/control-loop.png) #### The Controller sync method The Controller [sync](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/controller.go#L663) method is called by the _Workqueue_ to process a change of a resource. The method determines the _kind_ of the resource and calls the appropriate _sync_ method (Such as _syncIngress_ for Ingress resources). To explain how the sync methods work, we will examine the most important one: the _syncIngress_ method, and describe how it processes a new Ingress resource. ![image](/nic/controller-sync.png) 1. The _Workqueue_ calls the _sync_ method and passes a workqueue element to it that includes the changed resource _kind_ and _key_ (The key is the resource namespace/name such as “default/cafe-ingress”). 1. Using the _kind_, the _sync_ method calls the appropriate sync method and passes the resource key. For Ingress resources, the method is _syncIngress_. 1. _syncIngress_ gets the Ingress resource from the *Ingress Store* using the key. The _Store_ is controlled by the _Ingress Informer_. In the code, we use the helper _storeToIngressLister_ type that wraps the _Store_. 1. _syncIngress_ calls _AddOrUpdateIngress_ of the _Configuration_, passing the Ingress along. The [Configuration](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/configuration.go#L320) is a component that represents a valid collection of load balancing configuration resources (Ingresses, VirtualServers, VirtualServerRoutes, TransportServers), ready to be converted to the NGINX configuration (see the [Configuration section](#configuration) for more details). _AddOrUpdateIngress_ returns a list of [ResourceChanges](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/configuration.go#L59), which must be reflected in the NGINX config. Typically, for a new Ingress resource, the _Configuration_ returns only a single _ResourceChange_. 1. _syncIngress_ calls _processChanges_, which processes the single Ingress _ResourceChange_. 1. _processChanges_ creates an extended Ingress resource (_IngressEx_) that includes the original Ingress resource and its dependencies, such as Endpoints and Secrets, to generate the NGINX configuration. For simplicity, we don’t show this step on the diagram. 1. _processChanges_ calls _AddOrUpdateIngress_ of the _Configurator_ and passes the extended Ingress resource. 1. _Configurator_ generates an NGINX configuration file based on the extended Ingress resource, then: 1. Calls _Manager’s CreateConfig()_ to update the config for the Ingress resource. 1. Calls _Manager’s Reload()_ to reload NGINX. 1. The reload status is propagated from _Manager_ to _processChanges_, and is either a success or a failure with an error message. 1. _processChanges_ calls _updateRegularIngressStatusAndEvent_ to update the status of the Ingress resource and emit an event with the status of the reload: both make an API call to the Kubernetes API. **Additional notes**: - Many details are not included for conciseness: the source code provides the most granular detail. - The _syncVirtualServer_, _syncVirtualServerRoute_, and _syncTransportServer_ methods are similar to _syncIngress_, while other sync methods are different. However, those methods typically find the affected Ingress, VirtualServer, and TransportServer resources and regenerate the configuration for them. - The _Workqueue_ has only a single worker thread that calls the sync method synchronously, meaning the _Control Loop_ processes only one change at a time. #### Helper components There are two additional helper components crucial for processing changes: _Configuration_ and _LocalSecretStore_. ##### Configuration [_Configuration_](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/configuration.go#L320) holds the latest valid state of the NGINX Ingress Controller LTS load balancing configuration resources: Ingresses, VirtualServers, VirtualServerRoutes, TransportServers, and GlobalConfiguration. The _Configuration_ supports add, update and delete operations on the resources. When you invoke these operations on a resource in the Configuration, it performs the following: 1. Validates the object (For add or update) 1. Calculates the changes to the affected resources that are necessary to propagate to the NGINX configuration, returning the changes to the caller. For example, when you add a new Ingress resource, the _Configuration_ returns a change requiring NGINX Ingress Controller LTS to add the configuration for that Ingress to the NGINX configuration files. If you made an existing Ingress resource invalid, the _Configuration_ returns a change requiring NGINX Ingress Controller LTS to remove the configuration for that Ingress from the NGINX configuration files. Additionally, the _Configuration_ ensures that only one Ingress/VirtualServer/TransportServer (TLS Passthrough) resource holds a particular host (For example, cafe.example.com) and only one TransportServer (TCP/UDP) holds a particular listener (Such as port 53 for UDP). This ensures that no host or listener collisions happen in the NGINX configuration. Ultimately, NGINX Ingress Controller LTS ensures the NGINX config on the filesystem reflects the state of the objects in the _Configuration_ at any point in time. ##### LocalSecretStore [_LocalSecretStore_](https://github.com/nginx/kubernetes-ingress/blob/v1.11.0/internal/k8s/secrets/store.go#L32) (of the _SecretStore_ interface) holds the valid Secret resources and keeps the corresponding files on the filesystem in sync with them. Secrets are used to hold TLS certificates and keys (type `kubernetes.io/tls`), CAs (`nginx.org/ca`), JWKs (`nginx.org/jwk`), and client secrets for an OIDC provider (`nginx.org/oidc`). When _Controller_ processes a change to a configuration resource like Ingress, it creates an extended version of a resource that includes the dependencies (Such as Secrets) necessary to generate the NGINX configuration. _LocalSecretStore_ allows _Controller_ to reference the filesystem for a secret using the secret key (namespace/name). ## Reloading NGINX The following sections describe how NGINX reloads and how NGINX Ingress Controller LTS specifically affects this process. ### How NGINX reloads work Reloading NGINX is necessary to apply new configuration changes and occurs with these steps: 1. The administrator sends a HUP (hangup) signal to the NGINX master process to trigger a reload. 1. The master process brings down the worker processes with the old configuration and starts worker processes with the new configuration. 1. The administrator verifies the reload has successfully finished. The [NGINX documentation](https://nginx.org/en/docs/control.html#reconfiguration) has more details about reloading. #### How to reload NGINX and confirm success The NGINX binary (`nginx`) supports the reload operation with the `-s reload` option. When you run this option: 1. It validates the new NGINX configuration and exits if it is invalid printing the error messages to the stderr. 1. It sends a HUP signal to the NGINX master process and exits. As an alternative, you can send a HUP signal to the NGINX master process directly. Once the reload operation has been invoked with `nginx -s reload`, there is no wait period for NGINX to finish reloading. This means it is the responsibility of an administator to check it is finished, for which there are a few options: - Check if the master process created new worker processes. Two ways are by running `ps` or reading the `/proc` file system. - Send an HTTP request to NGINX, to see if a new worker process responds. This signifies that NGINX reloaded successfully: this method requires additional NGINX configuration, explained below. NGINX reloads take roughly 200ms. The factors affecting reload time are configuration size and details, the number of TLS certificates/keys, enabled modules, and available CPU resources. #### Potential problems Most of the time, if `nginx -s reload` executes, the reload will also succeed. In the rare case a reload fails, the NGINX master process will print the an error message. This is an example: ```shell 2022/07/09 00:56:42 [emerg] 1353#1353: limit_req "one" uses the "$remote_addr" key while previously it used the "$binary_remote_addr" key ``` The operation is graceful; reloading doesn't lead to any traffic loss by NGINX. However, frequent reloads can lead to high memory utilization and potential OOM (Out-Of-Memory) errors, resulting in traffic loss. This can most likely happen if you (1) proxy traffic that utilizes long-lived connections (ex: Websockets, gRPC) and (2) reload frequently. In these scenarios, you can end up with multiple generations of NGINX worker processes that are shutting down which will force old workers to shut down after the timeout). Eventually, all those worker processes can exhaust the system's available memory. Old NGINX workers will not shut down until all connections are terminated either by clients or backends, unless you configure [worker_shutdown_timeout](https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout). Since both the old and new NGINX worker processes coexist during a reload, reloading can lead to two spikes in memory utilization. With a lack of available memory, the NGINX master process can fail to create new worker processes. ### Reloading in NGINX Ingress Controller LTS NGINX Ingress Controller LTS reloads NGINX to apply configuration changes. To facilitate reloading, NGINX Ingress Controller LTS configures a server listening on the Unix socket `unix:/var/lib/nginx/nginx-config-version.sock` that responds with the configuration version for `/configVersion` URI. NGINX Ingress Controller LTS writes the configuration to `/etc/nginx/config-version.conf`. Reloads occur with this sequence of steps: 1. NGINX Ingress Controller LTS updates generated configuration files, including any secrets. 1. NGINX Ingress Controller LTS updates the config version in `/etc/nginx/config-version.conf`. 1. NGINX Ingress Controller LTS runs `nginx -s reload`. If the command fails, NGINX Ingress Controller LTS logs the error and considers the reload failed. 1. If the command succeeds, NGINX Ingress Controller LTS periodically checks for the config version by sending an HTTP request to the config version server on `unix:/var/lib/nginx/nginx-config-version.sock`. 1. Once NGINX Ingress Controller LTS sees the correct config version returned by NGINX, it considers the reload successful. If it doesn't see the correct configuration version after the configurable timeout ( [`-nginx-reload-timeout`](/nic/lts/configuration/global-configuration/command-line-arguments)), NGINX Ingress Controller LTS considers the reload failed. The [NGINX Ingress Controller LTS Control Loop](#the-control-loop) stops during a reload so that it cannot affect configuration files or reload NGINX until the current reload succeeds or fails. ### When NGINX Ingress Controller LTS reloads NGINX NGINX Ingress Controller LTS reloads NGINX every time the Control Loop processes a change that affects the generated NGINX configuration. In general, every time a monitored resource is changed, NGINX Ingress Controller LTS will regenerate the configuration and reload NGINX. There are three special cases: - *Start*. When NGINX Ingress Controller LTS starts, it processes all resources in the cluster and only then reloads NGINX. This avoids a "reload storm" by reloading only once. - *Batch updates*. When NGINX Ingress Controller LTS receives a number of concurrent requests from the Kubernetes API, it will pause NGINX reloads until the task queue is empty. This reduces the number of reloads to minimize the impact of batch updates and reduce the risk of OOM (Out of Memory) errors. - *Endpoint updates*. NGINX Ingress Controller LTS uses the NGINX Plus API to update the corresponding upstreams for endpoint changes and skips reloading. -------------------------------------------------------------------------------- title: "Troubleshooting common issues" url: https://docs.nginx.com/nginx-ingress-controller/lts/troubleshooting/troubleshoot-common/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot common issues with NGINX Ingress Controller LTS. Instruction for specific resources is available in the [Troubleshooting](/nic/lts/troubleshooting/) section. ## Common issues | Problem Area | Symptom | Troubleshooting Method | Common Cause | |-----|-----|-----|-----| | Startup | NGINX Ingress Controller LTS fails to start. | Check the logs. | Misconfigured RBAC, a missing default server TLS Secret.| | Ingress resource and annotations | The configuration is not applied | Check the events of the Ingress resource, check the logs, check the generated config. | Invalid values of annotations. | | VirtualServer and VirtualServerRoute resources | The configuration is not applied. | Check the events of the VirtualServer and VirtualServerRoutes, check the logs, check the generated config. | VirtualServer or VirtualServerRoute is invalid. | | Policy resource | The configuration is not applied. | Check the events of the Policy resource as well as the events of the VirtualServers that reference that policy, check the logs, check the generated config. | Policy is invalid. | | ConfigMap keys | The configuration is not applied. | Check the events of the ConfigMap, check the logs, check the generated config. | Invalid values of ConfigMap keys. | | NGINX | NGINX responds with unexpected responses. | Check the logs, check the generated config, check the live activity dashboard, run NGINX in the debug mode. | Unhealthy backend pods, a misconfigured backend service. | ## Troubleshooting Methods This section explains how to gather additional information for troubleshooting. The commands examples make the following assumptions: - That NGINX Ingress Controller LTS is deployed in the namespace `nginx-ingress`. - `` is the name of one of the NGINX Ingress Controller LTS pods. ### Check NGINX Ingress Controller LTS logs To check NGINX Ingress Controller LTS logs, which include both information from NGINX Ingress Controller LTS and NGINX's access and error logs, run the following command: ```shell kubectl logs -n nginx-ingress ``` ### Check the generated configuration files For each Ingress/VirtualServer resource, NGINX Ingress Controller LTS generates a corresponding NGINX configuration file in the `/etc/nginx/conf.d folder`. Additionally, NGINX Ingress Controller LTS generates the main configuration file `/etc/nginx/nginx.conf`, which includes all the configurations files from `/etc/nginx/conf.d`. The configuration for a VirtualServerRoute resource is located in the configuration file of the VirtualServer that references the resource. You can view the content of the main configuration file by running: ```shell kubectl exec -n nginx-ingress -- cat /etc/nginx/nginx.conf ``` Similarly, you can view the content of any generated configuration file in the `/etc/nginx/conf.d` folder. You can also print all NGINX configuration files together: ```shell kubectl exec -n nginx-ingress -- nginx -T ``` However, this command will fail if any of the configuration files is not valid. ### Check the Live Activity Monitoring Dashboard The live activity monitoring dashboard shows the real-time information about NGINX Plus and the applications it is load balancing, which is helpful for troubleshooting. To access the dashboard, read the [Status Page](/nic/lts/logging-and-monitoring/status-page.md) topic. ### Enable debugging for NGINX Ingress Controller LTS For additional NGINX Ingress Controller LTS debugging, you can enable debug settings to get more verbose logging. Increasing the debug log levels for NGINX Ingress Controller LTS will also apply to NGINX itself. There are two places to configure more verbose logging for NGINX Ingress Controller LTS: 1. Command line arguments 1. Configmap settings **Command line arguments** When using `manifest` for deployment, use the command line argument `-nginx-debug` in your deployment, daemonset, or statefulset. You can add the `-log-level` parameter to increase the verbosity of the NGINX Ingress Controller LTS process. Here is a small snippet of setting these command line arguments in the `args` section of a deployment: ```yaml args: - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config - -enable-cert-manager - -nginx-debug - -log-level=debug ``` **ConfigMap settings** You can configure `error-log-level` in NGINX Ingress Controller LTS `configMap`: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data: error-log-level: "debug" ``` **Using Helm** If you are using `helm`, you can adjust these two settings: ```none controller.nginxDebug = true or false controller.loglevel = fatal, error, warn, info, debug or trace ``` For example, if using a `values.yaml` file: ```yaml ## Enables debugging for NGINX. Uses the nginx-debug binary. Requires error-log-level: debug in ConfigMap via `controller.config.entries`. nginxDebug: true ## The log level of the Ingress Controller. logLevel: debug ``` This is a more complete example `values.yaml` file: ```yaml controller: kind: Deployment nginxDebug: true logLevel: debug annotations: nginx: ingress-prod pod: annotations: prometheus.io/scrape: "true" prometheus.io/port: "9113" prometheus.io/scheme: http extraLabels: env: prod-weset nginxplus: plus image: repository: private-registry.nginx.com/nginx-ic/lts/nginx-plus-ingress tag: # NGINX Configmap config: entries: error-log-level: "debug" proxy_connet_timeout: "5s" http-snippets: | underscores_in_headers on; ingressClass: nginx ``` Enable the `nginx-debug` CLI argument and change the `error-log-level` to `debug` to capture more output for debugging. **note:** It is recommended to only enable `nginx-debug` CLI and the `error-log-level` when debugging. #### Example debug NGINX Ingress Controller LTS Output These logs show some of the additional entries when debugging is enabled for NGINX Ingress Controller LTS. ```shell I1026 15:39:03.269092 1 manager.go:301] Reloading nginx with configVersion: 1 I1026 15:39:03.269115 1 utils.go:17] executing /usr/sbin/nginx-debug -s reload -e stderr 2022/10/26 15:39:03 [notice] 19#19: signal 1 (SIGHUP) received from 42, reconfiguring 2022/10/26 15:39:03 [debug] 19#19: wake up, sigio 0 2022/10/26 15:39:03 [notice] 19#19: reconfiguring 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF0A420:16384 @16 2022/10/26 15:39:03 [debug] 19#19: add cleanup: 000056362AF0C318 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF48230:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF00DE0:4096 2022/10/26 15:39:03 [debug] 19#19: read: 46, 000056362AF00DE0, 3090, 0 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF58670:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF12440:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF13500:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF145C0:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5C680:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5D740:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5E800:4280 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF5F8C0:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF41500:4096 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF638D0:8192 2022/10/26 15:39:03 [debug] 19#19: include /etc/nginx/mime.types 2022/10/26 15:39:03 [debug] 19#19: include /etc/nginx/mime.types 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF658E0:4096 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF668F0:5349 2022/10/26 15:39:03 [debug] 19#19: read: 47, 000056362AF658E0, 4096, 0 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF67DE0:4096 2022/10/26 15:39:03 [debug] 19#19: read: 47, 000056362AF658E1, 1253, 4096 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF68DF0:16384 @16 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF6CE00:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF70E10:524288 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AFF0E20:524288 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362B070E30:524288 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362B0F0E40:400280 ``` Once you have completed your debugging process, you can change the values back to the original values. -------------------------------------------------------------------------------- title: "Troubleshooting common issues" url: https://docs.nginx.com/nginx-ingress-controller/troubleshooting/troubleshoot-common/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot common issues with NGINX Ingress Controller. Instruction for specific resources is available in the [Troubleshooting](/nic/troubleshooting/) section. ## Common issues | Problem Area | Symptom | Troubleshooting Method | Common Cause | |-----|-----|-----|-----| | Startup | NGINX Ingress Controller fails to start. | Check the logs. | Misconfigured RBAC, a missing default server TLS Secret.| | Ingress resource and annotations | The configuration is not applied | Check the events of the Ingress resource, check the logs, check the generated config. | Invalid values of annotations. | | VirtualServer and VirtualServerRoute resources | The configuration is not applied. | Check the events of the VirtualServer and VirtualServerRoutes, check the logs, check the generated config. | VirtualServer or VirtualServerRoute is invalid. | | Policy resource | The configuration is not applied. | Check the events of the Policy resource as well as the events of the VirtualServers that reference that policy, check the logs, check the generated config. | Policy is invalid. | | ConfigMap keys | The configuration is not applied. | Check the events of the ConfigMap, check the logs, check the generated config. | Invalid values of ConfigMap keys. | | NGINX | NGINX responds with unexpected responses. | Check the logs, check the generated config, check the live activity dashboard (NGINX Plus only), run NGINX in the debug mode. | Unhealthy backend pods, a misconfigured backend service. | ## Troubleshooting Methods This section explains how to gather additional information for troubleshooting. The commands examples make the following assumptions: - That NGINX Ingress Controller is deployed in the namespace `nginx-ingress`. - `` is the name of one of the NGINX Ingress Controller pods. ### Check NGINX Ingress Controller logs To check NGINX Ingress Controller logs, which include both information from NGINX Ingress Controller and NGINX's access and error logs, run the following command: ```shell kubectl logs -n nginx-ingress ``` ### Check the generated configuration files For each Ingress/VirtualServer resource, NGINX Ingress Controller generates a corresponding NGINX configuration file in the `/etc/nginx/conf.d folder`. Additionally, NGINX Ingress Controller generates the main configuration file `/etc/nginx/nginx.conf`, which includes all the configurations files from `/etc/nginx/conf.d`. The configuration for a VirtualServerRoute resource is located in the configuration file of the VirtualServer that references the resource. You can view the content of the main configuration file by running: ```shell kubectl exec -n nginx-ingress -- cat /etc/nginx/nginx.conf ``` Similarly, you can view the content of any generated configuration file in the `/etc/nginx/conf.d` folder. You can also print all NGINX configuration files together: ```shell kubectl exec -n nginx-ingress -- nginx -T ``` However, this command will fail if any of the configuration files is not valid. ### Check the Live Activity Monitoring Dashboard The live activity monitoring dashboard shows the real-time information about NGINX Plus and the applications it is load balancing, which is helpful for troubleshooting. To access the dashboard, read the [Status Page](/nic/logging-and-monitoring/status-page.md) topic. ### Enable debugging for NGINX Ingress Controller For additional NGINX Ingress Controller debugging, you can enable debug settings to get more verbose logging. Increasing the debug log levels for NGINX Ingress Controller will also apply to NGINX itself. There are two places to configure more verbose logging for NGINX Ingress Controller: 1. Command line arguments 1. Configmap settings **Command line arguments** When using `manifest` for deployment, use the command line argument `-nginx-debug` in your deployment, daemonset, or statefulset. You can add the `-log-level` parameter to increase the verbosity of the NGINX Ingress Controller process. Here is a small snippet of setting these command line arguments in the `args` section of a deployment: ```yaml args: - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config - -enable-cert-manager - -nginx-debug - -log-level=debug ``` **ConfigMap settings** You can configure `error-log-level` in NGINX Ingress Controller `configMap`: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data: error-log-level: "debug" ``` **Using Helm** If you are using `helm`, you can adjust these two settings: ```none controller.nginxDebug = true or false controller.loglevel = fatal, error, warn, info, debug or trace ``` For example, if using a `values.yaml` file: ```yaml ## Enables debugging for NGINX. Uses the nginx-debug binary. Requires error-log-level: debug in ConfigMap via `controller.config.entries`. nginxDebug: true ## The log level of the Ingress Controller. logLevel: debug ``` This is a more complete example `values.yaml` file: ```yaml controller: kind: Deployment nginxDebug: true logLevel: debug annotations: nginx: ingress-prod pod: annotations: prometheus.io/scrape: "true" prometheus.io/port: "9113" prometheus.io/scheme: http extraLabels: env: prod-weset nginxplus: plus image: repository: nginx/nginx-ingress tag: # NGINX Configmap config: entries: error-log-level: "debug" proxy_connet_timeout: "5s" http-snippets: | underscores_in_headers on; ingressClass: nginx ``` Enable the `nginx-debug` CLI argument and change the `error-log-level` to `debug` to capture more output for debugging. **Note:** It is recommended to only enable `nginx-debug` CLI and the `error-log-level` when debugging. #### Example debug NGINX Ingress Controller Output These logs show some of the additional entries when debugging is enabled for NGINX Ingress Controller. ```shell I1026 15:39:03.269092 1 manager.go:301] Reloading nginx with configVersion: 1 I1026 15:39:03.269115 1 utils.go:17] executing /usr/sbin/nginx-debug -s reload -e stderr 2022/10/26 15:39:03 [notice] 19#19: signal 1 (SIGHUP) received from 42, reconfiguring 2022/10/26 15:39:03 [debug] 19#19: wake up, sigio 0 2022/10/26 15:39:03 [notice] 19#19: reconfiguring 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF0A420:16384 @16 2022/10/26 15:39:03 [debug] 19#19: add cleanup: 000056362AF0C318 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF48230:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF00DE0:4096 2022/10/26 15:39:03 [debug] 19#19: read: 46, 000056362AF00DE0, 3090, 0 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF58670:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF12440:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF13500:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF145C0:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5C680:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5D740:4280 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF5E800:4280 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF5F8C0:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF41500:4096 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF638D0:8192 2022/10/26 15:39:03 [debug] 19#19: include /etc/nginx/mime.types 2022/10/26 15:39:03 [debug] 19#19: include /etc/nginx/mime.types 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF658E0:4096 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF668F0:5349 2022/10/26 15:39:03 [debug] 19#19: read: 47, 000056362AF658E0, 4096, 0 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF67DE0:4096 2022/10/26 15:39:03 [debug] 19#19: read: 47, 000056362AF658E1, 1253, 4096 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF68DF0:16384 @16 2022/10/26 15:39:03 [debug] 19#19: posix_memalign: 000056362AF6CE00:16384 @16 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AF70E10:524288 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362AFF0E20:524288 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362B070E30:524288 2022/10/26 15:39:03 [debug] 19#19: malloc: 000056362B0F0E40:400280 ``` Once you have completed your debugging process, you can change the values back to the original values. -------------------------------------------------------------------------------- title: "View the NGINX status page" url: https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/status-page/index.md -------------------------------------------------------------------------------- This document explains how to get access to the stub status in NGINX and the dashboard in NGINX Plus. NGINX comes with a status page that reports basic metrics about NGINX called the [stub status](https://nginx.org/en/docs/http/ngx_http_stub_status_module.html). NGINX Plus comes with a [dashboard](/nginx/admin-guide/monitoring/live-activity-monitoring.md) that reports key load-balancing and performance metrics. F5 DoS for NGINX comes with a [dashboard](/nap-dos/monitoring/live-activity-monitoring.md) that shows the status and information of the protected objects. This doc shows how to get access to the stub status/dashboard. ## Accessing Stub Status Prerequisites: 1. The stub status is enabled by default. Ensure that the `nginx-status` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md) is not set to false. 1. The stub status is available on port 8080 by default. It is customizable by the `nginx-status-port` command-line argument. If yours is not on 8080, modify the kubectl proxy command below. To access the status: 1. Use the `kubectl port-forward` command to forward connections to port 8080 on your local machine to port 8080 of an NGINX Ingress Controller pod (replace `` with the actual name of a pod):. ```shell kubectl port-forward 8080:8080 --namespace=nginx-ingress ``` 1. Open your browser at [http://127.0.0.1:8080/stub_status](http://127.0.0.1:8080/stub_status) to access the status. If you want to access the stub status externally (without `kubectl port-forward`): 1. Configure `-nginx-status-allow-cidrs` command-line argument with IP/CIDR blocks for which you want to allow access to the status. By default, the access is allowed for `127.0.0.1,::1`. 1. Use the IP/port through which the Ingress Controller pod/pods are available to connect the stub status at the `/stub_status` path. ## Accessing Live Activity Monitoring Dashboard Prerequisites: 1. The dashboard is enabled by default. Ensure that the `nginx-status` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md) is not set to false. 1. The dashboard is available on port 8080 by default. It is customizable by the `nginx-status-port` command-line argument. If yours is not on 8080, modify the kubectl proxy command below. To access the dashboard: 1. Use the `kubectl port-forward` command to forward connections to port 8080 on your local machine to port 8080 of an NGINX Plus Ingress Controller pod (replace `` with the actual name of a pod): ```shell kubectl port-forward 8080:8080 --namespace=nginx-ingress ``` 1. Open your browser at to access the dashboard. 1. App Protect DoS: Open your browser at to access the dashboard. If you want to access the dashboard externally (without `kubectl port-forward`): 1. Configure `-nginx-status-allow-cidrs` command-line argument with IP/CIDR blocks for which you want to allow access to the dashboard. By default, the access is allowed for `127.0.0.1,::1`. 1. Use the IP/port through which the Ingress Controller pod/pods are available to connect the dashboard at the `/dashboard.html` path. **Note**: The [API](https://nginx.org/en/docs/http/ngx_http_api_module.html), which the dashboard uses to get the metrics, is also accessible: use the `/api` path. For App Protect DoS use the `/api/dos` path. Note that the API is configured in the read-only mode. -------------------------------------------------------------------------------- title: "View the NGINX status page" url: https://docs.nginx.com/nginx-ingress-controller/lts/logging-and-monitoring/status-page/index.md -------------------------------------------------------------------------------- This document explains how to get access to the dashboard in NGINX Plus. NGINX Plus comes with a [dashboard](/nginx/admin-guide/monitoring/live-activity-monitoring.md) that reports key load-balancing and performance metrics. ## Accessing Live Activity Monitoring Dashboard Prerequisites: 1. The dashboard is enabled by default. Ensure that the `nginx-status` [command-line argument](/nic/lts/configuration/global-configuration/command-line-arguments.md) is not set to false. 1. The dashboard is available on port 8080 by default. It is customizable by the `nginx-status-port` command-line argument. If yours is not on 8080, modify the kubectl proxy command below. To access the dashboard: 1. Use the `kubectl port-forward` command to forward connections to port 8080 on your local machine to port 8080 of an NGINX Plus Ingress Controller pod (replace `` with the actual name of a pod): ```shell kubectl port-forward 8080:8080 --namespace=nginx-ingress ``` 1. Open your browser at to access the dashboard. If you want to access the dashboard externally (without `kubectl port-forward`): 1. Configure `-nginx-status-allow-cidrs` command-line argument with IP/CIDR blocks for which you want to allow access to the dashboard. By default, the access is allowed for `127.0.0.1,::1`. 1. Use the IP/port through which the Ingress Controller pod/pods are available to connect the dashboard at the `/dashboard.html` path. **Note**: The [API](https://nginx.org/en/docs/http/ngx_http_api_module.html), which the dashboard uses to get the metrics, is also accessible: use the `/api` path. For App Protect DoS use the `/api/dos` path. Note that the API is configured in the read-only mode. -------------------------------------------------------------------------------- title: "Upgrade NGINX Ingress Controller to F5 WAF for NGINX with PLM" url: https://docs.nginx.com/nginx-ingress-controller/install/plm-upgrade/index.md -------------------------------------------------------------------------------- Use this guide to upgrade an existing F5 NGINX Ingress Controller + F5 WAF for NGINX deployment from in-pod App Protect policy compilation to Policy Lifecycle Management (PLM). PLM compiles `APPolicy` and `APLogConf` resources in a dedicated controller and stores the resulting bundles in an in-cluster S3-compatible object store. NGINX Ingress Controller then fetches the compiled bundles instead of compiling them in the data plane. Under PLM, the fields you already write on the NGINX Ingress Controller `Policy` resource (`waf.apPolicy` and `waf.securityLogs[].apLogConf`) continue to work unchanged. You don't need to rewrite the `k8s.nginx.org/v1` Policy manifest for a VirtualServer or Ingress during the upgrade. By the end of this guide, you'll have: - The PLM backend installed alongside your existing NGINX Ingress Controller deployment. - NGINX Ingress Controller upgraded to a PLM-capable release with PLM storage configured. - Existing `APPolicy` and `APLogConf` resources adopted by PLM and compiled into bundles. - WAF-protected traffic served by NGINX Ingress Controller, with bundles fetched from PLM storage instead of compiled in the data plane. ## Before you begin Before you start, make sure you have: - An existing NGINX Ingress Controller + F5 WAF for NGINX deployment running in your cluster. - Existing `APPolicy` and `APLogConf` resources served by the `appprotect.f5.com/v1beta1` CRDs shipped with your current NGINX Ingress Controller installation. - `kubectl` and Helm access to the cluster. - Credentials for `private-registry.nginx.com`. Record your current values before you begin: | Value | Where to find it | |---|---| | NGINX Ingress Controller release name | `helm list --namespace ` | | NGINX Ingress Controller chart version | `helm list --namespace ` | | Existing NGINX Ingress Controller values | `helm get values --namespace ` | | Existing `APPolicy` resources | `kubectl get appolicy --all-namespaces` | | Existing `APLogConf` resources | `kubectl get aplogconf --all-namespaces` | This guide uses `nic` as the NGINX Ingress Controller release name, `nginx-ingress` as the NGINX Ingress Controller namespace, `plm-system` as the PLM namespace, and `plm` as the PLM release name. Replace these with your own values consistently throughout. ## Deploy PLM infrastructure Install the PLM backend into the `plm-system` namespace. The install adds the `appprotect.f5.com/v1` versions to the existing `appprotect.f5.com` CRDs. Because the v1 versions are a superset of v1beta1, adding them doesn't affect NGINX Ingress Controller while it still watches v1beta1. The Policy Lifecycle Manager (PLM) backend runs as a Kubernetes operator. It watches WAF custom resources and compiles WAF policies into bundles. The Policy Controller delegates compilation to a separate compiler service over gRPC. The resulting bundles are stored in an embedded SeaweedFS S3-compatible object store. F5 WAF for NGINX is installed using a separate Helm chart from your NGINX data plane. The steps in this section install only the F5 WAF for NGINX PLM components and do not affect your existing NGINX installation. ### Create the registry pull secret Create a namespace for the PLM components, store your JWT in a Kubernetes Secret, then create the registry pull secret for the private F5 container registry. 1. Create the namespace and store your JWT. The following commands assume your JWT file is named `license.jwt`: ```shell kubectl create namespace plm-system kubectl create secret generic jwt-reg-secret \ --namespace plm-system \ --from-file=license.jwt ``` 2. Retrieve the JWT from the Secret and create the registry pull secret: ```shell JWT=$(kubectl get secret jwt-reg-secret \ --namespace plm-system \ -o jsonpath='{.data.license\.jwt}' | base64 -d) kubectl create secret docker-registry regcred \ --namespace plm-system \ --docker-server=private-registry.nginx.com \ --docker-username="$JWT" \ --docker-password=none \ --dry-run=client --output yaml | kubectl apply -f - ``` ### Install the Policy Controller Create a values file for the Helm installation. The `securityUpdatesRepo.cert` and `securityUpdatesRepo.key` fields are optional. They are only required if your signature repository needs certificate-based authentication. The Policy Controller starts successfully with these fields left empty. If your signature repository requires them, replace `` and `` with the base64-encoded contents of your `nginx-repo.crt` and `nginx-repo.key` files. To encode them, run: ```shell base64 --wrap=0 < nginx-repo.crt base64 --wrap=0 < nginx-repo.key ``` Create `/tmp/plm-values.yaml`: ```yaml imagePullSecrets: - name: regcred securityUpdatesRepo: cert: "" # optional: only needed for authenticated signature repository access key: "" # optional: only needed for authenticated signature repository access policyController: image: tag: "" compiler: image: tag: "" seaweedfsOperatorConfig: seaweedfs: image: tag: "" seaweedfs-operator: image: tag: "" pullSecrets: regcred ``` #### Enable TLS for PLM storage (optional) By default, communication between PLM components and the SeaweedFS object store uses unencrypted HTTP. To enable TLS, add a `certificates` block to `/tmp/plm-values.yaml`: ```yaml seaweedfsOperatorConfig: seaweedfs: certificates: enabled: true ``` **Note:** The PLM chart does not generate certificates. You must create the five Secrets listed in the commands below before running `helm upgrade --install`. If any Secret is missing, the SeaweedFS pods will fail to mount their certificates and will not start. **Note:** If you're enabling TLS on an existing installation, the storage backend restarts and objects written before the switch can become orphaned. See the [APPolicy shows `invalid` with `unexpected EOF` after enabling TLS](#troubleshoot-the-deployment) entry in the troubleshooting section. A fresh installation with TLS enabled from the start doesn't have this issue. Create the Secrets from your CA and certificate files before installing. The chart expects Secret names in the form `-f5-waf-seaweedfs-` — for the `plm` release name used in this tutorial, those are: ```shell kubectl create secret generic plm-f5-waf-seaweedfs-ca-cert \ --namespace plm-system \ --from-file=tls.crt= \ --from-file=ca.crt= kubectl create secret tls plm-f5-waf-seaweedfs-master-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-volume-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-filer-cert \ --namespace plm-system \ --cert= \ --key= kubectl create secret tls plm-f5-waf-seaweedfs-client-cert \ --namespace plm-system \ --cert= \ --key= ``` The CA Secret requires both `tls.crt` and `ca.crt` keys, both pointing to the same CA certificate file. The PLM chart mounts the CA using `tls.crt` into the Policy Controller, compiler, and SeaweedFS pods. The data plane's S3 client reads `ca.crt` from the same Secret when verifying the storage endpoint. The four component Secrets use `kubectl create secret tls`, which produces `tls.crt` and `tls.key` — no `ca.crt` key is needed for them. Replace each `` placeholder with the path to the corresponding certificate and key file from your PKI. The CA must sign all component certificates. If you don't have an existing PKI, generate a CA and sign the five component certificates before proceeding. #### Install the chart Add the NGINX Helm repository and install the chart: ```shell helm repo add nginx-stable https://helm.nginx.com/stable helm repo update nginx-stable helm upgrade --install plm nginx-stable/f5-waf-policy-controller \ --version \ --namespace plm-system \ --values /tmp/plm-values.yaml ``` To see all available configuration options for the PLM chart, run: ```shell helm show values nginx-stable/f5-waf-policy-controller --version ``` ### Verify the deployment Wait for all PLM components to become ready. The Policy Controller's init container waits for both the compiler service and the SeaweedFS S3 endpoint to be available before it starts, so the controller pod will show `Init:0/1` until SeaweedFS is ready. Wait for the SeaweedFS storage backend: ```shell kubectl rollout status deployment/plm-seaweedfs-operator \ --namespace plm-system --timeout=120s ``` The SeaweedFS operator creates the SeaweedFS pods after it reconciles the SeaweedFS custom resource, so there is a window where the operator deployment is ready but no SeaweedFS pods exist yet. Poll until the pods appear and are ready: ```shell end=$((SECONDS + 300)) until kubectl wait pods \ --selector app.kubernetes.io/name=seaweedfs \ --for=condition=Ready \ --namespace plm-system \ --timeout=10s 2>/dev/null; do if [ $SECONDS -ge $end ]; then echo "Timed out waiting for SeaweedFS pods" exit 1 fi sleep 5 done ``` Wait for the Policy Controller: ```shell kubectl rollout status deployment/plm-f5-waf-policy-controller \ --namespace plm-system --timeout=180s ``` Confirm all pods are running: ```shell kubectl get pods --namespace plm-system ``` Example output: ```text NAME READY STATUS RESTARTS plm-f5-waf-compiler-service-xxxxx 1/1 Running 0 plm-f5-waf-policy-controller-xxxxx 1/1 Running 0 plm-seaweedfs-operator-xxxxx 1/1 Running 0 plm-f5-waf-seaweed-master-0 1/1 Running 0 plm-f5-waf-seaweed-filer-0 1/1 Running 0 plm-f5-waf-seaweed-volume-0 1/1 Running 0 plm-f5-waf-seaweed-volume-1 1/1 Running 0 plm-f5-waf-seaweed-volume-2 1/1 Running 0 ``` Confirm the CRDs are present: ```shell kubectl get crd | grep appprotect.f5.com ``` Expected output: ```text aplogconfs.appprotect.f5.com appolicies.appprotect.f5.com apsignatures.appprotect.f5.com apusersigs.appprotect.f5.com ``` All eight pods running and all four CRDs present confirms the PLM backend is ready. ### Update the CRDs **Note:** Skip this step on a fresh install — Helm installs the CRDs automatically. Only follow these steps when upgrading an existing PLM installation. When upgrading PLM, apply the CRDs manually before running `helm upgrade`: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/waf-policy-controller//manifests/1-deploy-crds.yaml ``` ### Troubleshoot the deployment These are the most common failures during PLM installation, roughly in order of likelihood. #### Pods stuck in `ImagePullBackOff` The JWT is wrong, expired, or contains a line break. Check the events log: ```shell kubectl get events --namespace plm-system --field-selector reason=Failed ``` Use the full JWT string as the registry username. Use the literal string `none` as the password. #### Policy Controller stuck in `Init:0/1` The `Init:0/1` state is expected during startup. The init container waits for the compiler service and the S3 endpoint before it starts. If the pod stays in `Init:0/1` for more than a few minutes, check that the SeaweedFS pods are `Running`: ```shell kubectl get pods --namespace plm-system --selector app.kubernetes.io/name=seaweedfs ``` The most common cause is PVCs stuck in `Pending` because the cluster has no default StorageClass. #### SeaweedFS pods `Pending` SeaweedFS pods stay `Pending` when the cluster has no default StorageClass or insufficient capacity. Check the PVCs and available storage classes: ```shell kubectl get pvc --namespace plm-system kubectl get storageclass ``` #### `APPolicy` shows `invalid` with `unexpected EOF` after enabling TLS Enabling TLS on an existing installation restarts the storage backend. Objects written before TLS was enabled can become orphaned. Check the filer log: ```shell kubectl logs --namespace plm-system plm-f5-waf-seaweed-filer-0 | grep "not found" ``` If the output contains `volume N not found`, orphaned objects exist. Delete the affected `APPolicy` resource and reapply it. The Policy Controller regenerates the bundle. #### Helm install fails on a ClusterRole If the error references `seaweed-editor-role` or `seaweed-viewer-role`, another PLM installation already exists in the cluster. Only one PLM installation is supported per cluster. Remove the existing release before installing. #### Check the Policy Controller logs Use the Policy Controller logs to diagnose any policy-related failure: ```shell kubectl logs --namespace plm-system deploy/plm-f5-waf-policy-controller -c policy-controller ``` **Note:** The `-c policy-controller` flag is required because the pod has more than one container. The containers are distroless, so `kubectl exec` isn't available for interactive debugging. After the install completes, PLM adopts every existing `APPolicy` and `APLogConf` resource in the cluster. It does this by adding the `appprotect.f5.com/finalizer` finalizer and compiling each resource against its current signature package. Your running NGINX Ingress Controller continues to compile the same resources in-pod. The two mechanisms operate independently until you upgrade NGINX Ingress Controller. Confirm PLM has compiled the existing resources: ```shell kubectl get appolicy --all-namespaces \ --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.bundle.state' kubectl get aplogconf --all-namespaces \ --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.bundle.state' ``` Every resource should report `STATE: ready`. If any resource isn't `ready`, inspect the PLM policy-controller logs, resolve the compilation errors, and then continue. ## Look up the PLM storage endpoint and credentials NGINX Ingress Controller connects to PLM's SeaweedFS filer over S3 to fetch compiled bundles. Before you run `helm upgrade`, collect these four values from the PLM installation: 1. **PLM storage URL**: the SeaweedFS filer endpoint (HTTPS or HTTP). 2. **Credentials Secret**: the S3 credentials Secret. The access key ID is `admin` by default. The secret access key is in the `seaweedfs_admin_secret` field. 3. **CA Secret** (HTTPS only): verifies the SeaweedFS filer certificate. 4. **Client TLS Secret** (mutual TLS only): presented by NGINX Ingress Controller when it connects to the filer. List the Services PLM created and identify the filer: ```shell kubectl get service --namespace plm-system ``` Expected output includes an entry similar to: ```text NAME TYPE CLUSTER-IP PORT(S) plm-f5-waf-seaweed-filer ClusterIP 10.0.0.10 8333/TCP,9333/TCP,... ``` Assemble the URL from the service name, namespace, and port. Use `9333` for HTTPS and `8333` for HTTP: - HTTPS (mTLS): `https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333` - HTTP: `http://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:8333` List the Secrets PLM created: ```shell kubectl get secret --namespace plm-system ``` The default PLM install creates three Secrets that NGINX Ingress Controller references: - `plm-f5-waf-seaweedfs-auth`: SeaweedFS credentials. - `plm-f5-waf-seaweedfs-ca-cert`: CA certificate for the HTTPS filer. - `plm-f5-waf-seaweedfs-client-cert`: client TLS certificate for mTLS. Record the Secret references in `/` form. When you run `helm upgrade`, pass all four values to NGINX Ingress Controller using `--set controller.appprotect.plmStorage.*` flags. ## Apply the NGINX Ingress Controller CRDs Before you run `helm upgrade`, apply the NGINX Ingress Controller CRDs from the bundled manifest for your target release. The `deploy/crds.yaml` bundle contains every CRD the controller needs (`VirtualServer`, `VirtualServerRoute`, `Policy`, `TransportServer`, `GlobalConfiguration`, `DNSEndpoint`). The bundle deliberately excludes the App Protect CRDs, which PLM owns. ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` ## Section 1: Upgrade a VirtualServer-based deployment Follow this section if your existing NGINX Ingress Controller deployment routes traffic through `k8s.nginx.org/v1` VirtualServer resources that reference a Policy resource with `waf.apPolicy` and `waf.securityLogs[].apLogConf` fields. ### Upgrade NGINX Ingress Controller with PLM storage Upgrade the NGINX Ingress Controller release. `--reuse-values` preserves your existing configuration. The `--set` flags overlay PLM storage on top. `--skip-crds` prevents Helm from touching CRDs, because you applied the NGINX Ingress Controller CRDs in the previous step and PLM owns the App Protect CRDs. This example uses HTTPS PLM storage with mutual TLS. For HTTP storage, set `controller.appprotect.plmStorage.url` to `http://:8333` and omit the `caSecret` and `clientSSLSecret` flags. ```shell helm upgrade nic nginx-stable/nginx-ingress \ --namespace nginx-ingress \ --skip-crds \ --reuse-values \ --set controller.appprotect.enable=true \ --set controller.appprotect.v5=true \ --set controller.image.repository="private-registry.nginx.com/nginx-ic-nap-v5/nginx-plus-ingress" \ --set controller.image.tag="" \ --set controller.appprotect.plmStorage.url="https://plm-f5-waf-seaweed-filer.plm-system.svc.cluster.local:9333" \ --set controller.appprotect.plmStorage.credentialsSecret="plm-system/plm-f5-waf-seaweedfs-auth" \ --set controller.appprotect.plmStorage.caSecret="plm-system/plm-f5-waf-seaweedfs-ca-cert" \ --set controller.appprotect.plmStorage.clientSSLSecret="plm-system/plm-f5-waf-seaweedfs-client-cert" \ --set controller.appprotect.plmStorage.insecureSkipVerify=false ``` Wait for the rollout to complete: ```shell kubectl rollout status deployment/nic-nginx-ingress-controller \ --namespace nginx-ingress \ --timeout=180s ``` Confirm NGINX Ingress Controller is watching the v1 CRDs by inspecting the controller log: ```shell kubectl logs deployment/nic-nginx-ingress-controller \ --namespace nginx-ingress \ --container nginx-ingress | grep 'appprotect.f5.com/v' ``` Expected output includes: ```text Using appprotect.f5.com/v1 CRDs ``` ### Verify Policy status Check each WAF Policy referenced by a VirtualServer: ```shell kubectl get policy --all-namespaces \ --output custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATE:.status.state,REASON:.status.reason' ``` Expected output after the first successful fetch: ```text NAMESPACE NAME STATE REASON default waf-policy Valid AddedOrUpdated ``` If a Policy remains in `Warning` with `BundlePending`, the referenced `APPolicy` or `APLogConf` isn't yet `ready` in PLM. Confirm that PLM has compiled the resource, then retry. ### Verify traffic Send a normal request to the VirtualServer and confirm the application responds: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \ http://webapp.example.com:$IC_HTTP_PORT/ ``` Then send a request that triggers the configured WAF violation and confirm the response is `Request Rejected`. ### Confirm bundles come from PLM storage Check the bundle files in the ingress controller pod: ```shell NIC_POD=$(kubectl get pods --namespace nginx-ingress \ --selector app.kubernetes.io/name=nginx-ingress \ --output jsonpath='{.items[0].metadata.name}') kubectl exec --namespace nginx-ingress $NIC_POD --container nginx-ingress -- \ ls -l /etc/app_protect/bundles/ ``` Files named `fetched___policy.tgz` and `fetched___log_.tgz` confirm that the bundles were fetched from PLM storage. ## Section 2: Upgrade an Ingress-based deployment Follow this section if your existing NGINX Ingress Controller deployment routes traffic through Kubernetes Ingress resources. The procedure is the same as Section 1, with one additional Ingress-specific step. ### Audit Ingress annotations Under PLM, NGINX Ingress Controller doesn't support the App Protect Ingress annotations: - `appprotect.f5.com/app-protect-policy` - `appprotect.f5.com/app-protect-security-log` - `appprotect.f5.com/app-protect-security-log-enable` An Ingress that uses these annotations is accepted but produces a warning after the upgrade. WAF isn't applied to that route. Before you turn on PLM storage, migrate every such Ingress to a `k8s.nginx.org/v1` Policy resource. ### Upgrade NGINX Ingress Controller with PLM storage Run the same `helm upgrade` command shown in [Section 1, step 1](#1-upgrade-nginx-ingress-controller-with-plm-storage). ### Verify Ingress traffic For each Ingress, send a normal request and confirm the application responds. Then send a request that matches a WAF violation and confirm the response is `Request Rejected`. ### Confirm bundles come from PLM storage Use the same procedure as [Section 1, step 4](#4-confirm-bundles-come-from-plm-storage). ## Troubleshooting - **Policy stays in `BundlePending` after the upgrade.** The referenced `APPolicy` or `APLogConf` isn't `ready` in PLM. Run `kubectl describe appolicy ` and inspect the PLM policy-controller logs. - **NGINX Ingress Controller reports the referenced namespace isn't watched.** If the deployment sets `controller.watchNamespace`, include the namespace of every `APPolicy` and `APLogConf` resource. Also include the PLM namespace in `controller.watchSecretNamespace` so NGINX Ingress Controller can observe storage Secret rotation. - **Helm upgrade fails with a CRD conflict.** Confirm PLM is installed and its v1 CRDs are present, then run the upgrade with `--skip-crds`. -------------------------------------------------------------------------------- title: "Apply policies to resources" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/applying-policies/index.md -------------------------------------------------------------------------------- You can reference policies from VirtualServer, VirtualServerRoute, and Ingress resources. How you attach a policy depends on the resource type. ## VirtualServer You can attach policies at: - `spec.policies` for server-wide behavior - `spec.routes[].policies` for route-specific behavior Example: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe spec: host: cafe.example.com policies: - name: access-policy upstreams: - name: coffee service: coffee-svc port: 80 routes: - path: /coffee policies: - name: route-cors-policy action: pass: coffee ``` ## VirtualServerRoute You can attach policies at: - `spec.subroutes[].policies` Example: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: tea spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 subroutes: - path: /tea policies: - name: subroute-policy action: pass: tea ``` ## Ingress Ingress uses annotations instead of a `policies` field. Supported annotations are: - `nginx.org/policies` - `nginx.com/policies` Example: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webapp annotations: nginx.org/policies: access-policy,cors-policy spec: ingressClassName: nginx rules: - host: webapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: webapp port: number: 80 ``` ## What's next Learn about [ingress-specific policy behavior](/nic/configuration/policy-resource/ingress-specific-behavior.md). -------------------------------------------------------------------------------- title: "Compile F5 WAF for NGINX policies using NGINX Instance Manager" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-waf-v5/compile-waf-policies/index.md -------------------------------------------------------------------------------- ## Overview This guide describes how to use F5 NGINX Instance Manager to compile F5 WAF for NGINX Policies for use with NGINX Ingress Controller. F5 WAF for NGINX uses policies to configure which security features are set. When these policies are changed, they need to be compiled so that the engine can begin to use them. Compiling policies can take a large amount of time and resources. You can do this with the NGINX Instance Manager. This reduces the impact on a NGINX Ingress Controller deployment. By using NGINX Instance Manager to compile WAF policies, the policy bundle can also be used immediately by NGINX Ingress Controller without reloading. **Note:** NGINX Ingress Controller can fetch compiled bundles directly from NGINX Instance Manager or NGINX One Console without the manual download and upload steps described on this page. See [Connect F5 WAF for NGINX to bundle sources](/nic/integrations/app-protect-waf-v5/bundle-sources.md) for details on automated bundle fetching with optional polling for updates. The following steps describe how to use the NGINX Instance Manager API to create a new security policy, compile a bundle, then add it to NGINX Ingress Controller. ## Before you begin - A working [NGINX Instance Manager](/nim/deploy/) instance. - An [NGINX Instance Manager user](/nim/admin-guide/rbac/overview-rbac.md) for API requests. - A NGINX Ingress Controller [deployment with F5 WAF for NGINX](/nic/integrations/app-protect-waf/installation.md). ## Create a new security policy **Note:** You can skip this step if you intend to use an existing security policy. Create a [new security policy](/nim/waf-integration/policies-and-logs/policies/create-policy.md) using the API: this will require the use of a tool such as [`curl`](https://curl.se/) or [Postman](https://www.postman.com/) Create the file `simple-policy.json` with the contents below: ```json { "metadata": { "name": "Nginxbundletest", "displayName": "Nginxbundletest", "description": "Ignore cross-site scripting is a security policy that intentionally ignores cross site scripting." }, "content": "ewoJInBvbGljeSI6IHsKCQkibmFtZSI6ICJzaW1wbGUtYmxvY2tpbmctcG9saWN5IiwKCQkic2lnbmF0dXJlcyI6IFsKCQkJewoJCQkJInNpZ25hdHVyZUlkIjogMjAwMDAxODM0LAoJCQkJImVuYWJsZWQiOiBmYWxzZQoJCQl9CgkJXSwKCQkidGVtcGxhdGUiOiB7CgkJCSJuYW1lIjogIlBPTElDWV9URU1QTEFURV9OR0lOWF9CQVNFIgoJCX0sCgkJImFwcGxpY2F0aW9uTGFuZ3VhZ2UiOiAidXRmLTgiLAoJCSJlbmZvcmNlbWVudE1vZGUiOiAiYmxvY2tpbmciCgl9Cn0=" } ``` **Note:** The `content` value must be base64 encoded or you will encounter an error. Upload the policy JSON files with the API, which is the same method to create the bundle later. In the same directory you created `simple-policy.json`, create a POST request for NGINX Instance Manager using the API. ```shell curl -X POST https://{{NMS_FQDN}}/api/platform/v1/security/policies \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d @simple-policy.json ``` You should receive an API response similar to the following output, indicating the policy has been successfully created. ```json { "metadata": { "created": "2024-06-12T20:28:08.152171922Z", "description": "Ignore cross-site scripting is a security policy that intentionally ignores cross site scripting.", "displayName": "Nginxbundletest", "externalId": "", "externalIdType": "", "modified": "2024-06-12T20:28:08.152171922Z", "name": "Nginxbundletest", "revisionTimestamp": "2024-06-12T20:28:08.152171922Z", "uid": "6af9f261-658b-4be1-b07a-cebd83e917a1" }, "selfLink": { "rel": "/api/platform/v1/security/policies/6af9f261-658b-4be1-b07a-cebd83e917a1" } } ``` **Note:** Take note of the *uid* field: `"uid": "6af9f261-658b-4be1-b07a-cebd83e917a1"` It is one of two unique IDs we will use to download the bundle: it will be referenced as *policy-UID*. --- ## Create a new security bundle Once you have created (Or selected) a security policy, [create a security bundle](/nim/waf-integration/policies-and-logs/bundles/create-bundle.md) using the API. The version in the bundle you create **must** match the WAF compiler version you intend to use. You can check which version is installed in NGINX Instance Manager by checking the operating system packages. If the wrong version is noted in the JSON payload, you will receive an error similar to below: ```text {"code":13018,"message":"Error compiling the security policy set: One or more of the specified compiler versions does not exist. Check the compiler versions, then try again."} ``` Create the file `security-policy-bundles.json`: ```json { "bundles": [ { "appProtectWAFVersion": "", "policyName": "Nginxbundletest", "policyUID": "", "attackSignatureVersionDateTime": "latest", "threatCampaignVersionDateTime": "latest" } ] } ``` The *policyUID* value is left blank, as it is generated as part of the creating the bundle. Send a POST request to create the bundle through the API: ```shell curl -X POST https://{{NMS_FQDN}}/api/platform/v1/security/policies/bundles \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d @security-policy-bundles.json ``` You should receive a response similar to the following: ```json { "items": [ { "compilationStatus": { "message": "", "status": "compiling" }, "content": "", "metadata": { "appProtectWAFVersion": "", "attackSignatureVersionDateTime": "2024.02.21", "created": "2024-06-12T13:28:20.023775785-07:00", "modified": "2024-06-12T13:28:20.023775785-07:00", "policyName": "Nginxbundletest", "policyUID": "6af9f261-658b-4be1-b07a-cebd83e917a1", "threatCampaignVersionDateTime": "2024.02.25", "uid": "cbdf9577-6d81-43d6-8ce1-2e3d4714e8b5" } } ] } ``` You can use the API to list the security bundles, verifying the new addition: ```shell curl --location 'https://127.0.0.1/api/platform/v1/security/policies/bundles' \ -H "Authorization: Bearer " ``` ```json { "items": [ { "compilationStatus": { "message": "", "status": "compiled" }, "content": "", "metadata": { "appProtectWAFVersion": "", "attackSignatureVersionDateTime": "2024.02.21", "created": "2024-06-13T09:09:10.809-07:00", "modified": "2024-06-13T09:09:20-07:00", "policyName": "Nginxbundletest", "policyUID": "ec8681eb-1e25-4b71-93bd-b91f67c5ac99", "threatCampaignVersionDateTime": "2024.02.25", "uid": "de08b324-99d8-4155-b2eb-fe687b21034e" } } ] } ``` **Note:** Take note of the *uid* field: `"uid": "de08b324-99d8-4155-b2eb-fe687b21034e"` It is one of two unique IDs we will use to download the bundle: it will be referenced as *bundle-UID*. --- ## Download the security policy bundle Use a GET request to download the security bundle using the policy and bundle IDs: ```shell curl -X GET "https://{NMS_FQDN}/api/platform/v1/security/policies//bundles/" -H "Authorization: Bearer " | jq -r '.content' | base64 -d > security-policy-bundle.tgz ``` This GET request uses the policy and bundle IDs from the previous examples: ```shell curl -X GET -k 'https://127.0.0.1/api/platform/v1/security/policies/6af9f261-658b-4be1-b07a-cebd83e917a1/bundles/de08b324-99d8-4155-b2eb-fe687b21034e' \ -H "Authorization: Basic " \ | jq -r '.content' | base64 -d > security-policy-bundle.tgz ``` ## Download the security log bundle Use a GET request to download the `secops_dashboard` security log bundle. The security log bundle adjusts the format of the policy events to be compatible with NGINX Instance Manager: ```shell curl -X GET "https://{NMS_FQDN}/api/platform/v1/security/logprofiles/secops_dashboard//bundle" -H "Authorization: Bearer " | jq -r .compiledBundle | base64 -d > secops_dashboard.tgz ``` --- ## Add volumes and volumeMounts To use WAF security bundles, your NGINX Ingress Controller instance must have *volumes* and *volumeMounts*. Precise paths are used to detect when bundles are uploaded to the cluster. Here is an example of what to add: ```yaml volumes: - name: persistentVolumeClaim: claimName: volumeMounts: - name: mountPath: /etc/app_protect/bundles ``` A full example of a deployment file with `volumes` and `volumeMounts` could look like the following: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress namespace: nginx-ingress spec: replicas: 1 selector: matchLabels: app: nginx-ingress template: metadata: labels: app: nginx-ingress app.kubernetes.io/name: nginx-ingress #annotations: #prometheus.io/scrape: "true" #prometheus.io/port: "9113" #prometheus.io/scheme: http spec: serviceAccountName: nginx-ingress automountServiceAccountToken: true securityContext: seccompProfile: type: RuntimeDefault volumes: - name: nginx-bundle-mount emptydir: {} containers: - image: imagePullPolicy: IfNotPresent name: nginx-ingress ports: - name: http containerPort: 80 - name: https containerPort: 443 - name: readiness-port containerPort: 8081 - name: prometheus containerPort: 9113 readinessProbe: httpGet: path: /nginx-ready port: readiness-port periodSeconds: 1 resources: requests: cpu: "100m" memory: "128Mi" #limits: # cpu: "1" # memory: "1Gi" securityContext: allowPrivilegeEscalation: false runAsUser: 101 #nginx runAsNonRoot: true capabilities: drop: - ALL add: - NET_BIND_SERVICE volumeMounts: - name: bundle-mount mountPath: /etc/app_protect/bundles env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name args: - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config - -report-ingress-status - -external-service=nginx-ingress ``` --- ## Upload the security log bundle Upload the security log bundle binary file to the NGINX Ingress Controller pods. #### Helm ```shell kubectl cp /your/local/path/secops_dashboard.tgz /:etc/app_protect/bundles/secops_dashboard.tgz -c nginx-ingress ``` #### Manifests ```shell kubectl cp /your/local/path/secops_dashboard.tgz /:etc/app_protect/bundles/secops_dashboard.tgz -c nginx-plus-ingress ``` ## Upload the security policy bundle Upload the binary file to the NGINX Ingress Controller pods. #### Helm ```shell kubectl cp /your/local/path/.tgz /:etc/app_protect/bundles.tgz -c nginx-ingress ``` #### Manifests ```shell kubectl cp /your/local/path/.tgz /:etc/app_protect/bundles.tgz -c nginx-plus-ingress ``` --- ## Create WAF policy To process a bundle, you must create a new WAF policy. This policy is added to `/etc/app_protect/bundles`, allowing NGINX Ingress Controller to load it into WAF. The example below shows the required WAF policy, for the *apBundle* field you must use the [security bundle](#download-the-security-policy-bundle) binary file (a tarball). The *apLogBundle* field contains the `secops_dashboard.tgz` [file](#download-the-security-log-bundle). ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: spec: waf: enable: true apBundle: ".tgz" securityLogs: - enable: true apLogBundle: "secops_dashboard.tgz" logDest: "" ``` --- ## Create VirtualServer resource and apply policy Once the WAF policy has been created, link it to your *virtualServer resource*. ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: webapp spec: host: webapp.example.com policies: - name: upstreams: - name: webapp service: webapp-svc port: 80 routes: - path: / action: pass: webapp ``` Your `VirtualServer` should now apply the generated security policy to your traffic and emit security events to NGINX Instance Manager. -------------------------------------------------------------------------------- title: "ConfigMap resources" url: https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/index.md -------------------------------------------------------------------------------- When using F5 NGINX Ingress Controller, you can customize or fine tune NGINX behavior using ConfigMap resources. Examples include setting the number of worker processes or customizing the access log format. ## Using ConfigMap 1. The [Installation with Manifests](/nic/install/manifests.md) documentation deploy an empty ConfigMap while the default installation manifests specify it in the command-line arguments of the Ingress Controller. However, if you customized the manifests, to use ConfigMap, make sure to specify the ConfigMap resource to use the [command-line arguments](/nic/configuration/global-configuration/command-line-arguments) of NGINX Ingress Controller. 1. Create a ConfigMap file with the name *nginx-config.yaml* and set the values that make sense for your setup: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data: proxy-connect-timeout: "10s" proxy-read-timeout: "10s" client-max-body-size: "2m" ``` See the section [Summary of ConfigMap Keys](#configmap-keys) for the explanation of the available ConfigMap keys (such as `proxy-connect-timeout` in this example). 1. Create a new (or update the existing) ConfigMap resource: ```shell kubectl apply -f nginx-config.yaml ``` The NGINX configuration will be updated. --- ## ConfigMap and Ingress annotations ConfigMap applies globally, meaning that it affects every Ingress resource. In contrast, annotations always apply to their Ingress resource. Annotations can override some ConfigMap keys: an example is that the `nginx.org/proxy-connect-timeout` annotations overrides the `proxy-connect-timeout` ConfigMap key. For more information, view the [Advanced configuration with annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations) topic. ## ConfigMap and VirtualServer/VirtualServerRoute resources The ConfigMap affects every VirtualServer and VirtualServerRoute resources. However, the fields of those resources allow overriding some ConfigMap keys. For example, the `connect-timeout` field of the `upstream` overrides the `proxy-connect-timeout` ConfigMap key. For more information, view the [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources) topic. ## ConfigMap keys ### Ingress Controller (Unrelated to NGINX Configuration) |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*external-status-address* | Sets the address to be reported in the status of Ingress resources. Requires the *-report-status* command-line argument. Overrides the *-external-service* argument. | N/A | [Reporting resource status](/nic/configuration/global-configuration/reporting-resources-status) | ### General customization |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | |*proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | |*proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | |*client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | |*client-body-buffer-size* | Sets the value of the [client_body_buffer_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) directive. | N/A | | |*proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | |*proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | |*proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | |*proxy-busy-buffers-size* | Sets the value of the [proxy_busy_buffers_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) directive. | Depends on the platform. | | |*proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | |*set-real-ip-from* | Sets the value of the [set_real_ip_from](https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from) directive. | N/A | | |*real-ip-header* | Sets the value of the [real_ip_header](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header) directive. | *X-Real-IP* | | |*real-ip-recursive* | Enables or disables the [real_ip_recursive](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive) directive. | *False* | | |*default-server-return* | Configures the [return](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return) directive in the default server, which handles a client request if none of the hosts of Ingress or VirtualServer resources match. The default value configures NGINX to return a 404 error page. You can configure a fixed response or a redirect. For example, *default-server-return: 302 https://nginx.org* will redirect a client to *https://nginx.org*. When [-allow-empty-ingress-host](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/command-line-arguments/#cmdoption-allow-empty-ingress-host) is enabled and an empty-host Ingress is active, this directive applies to requests that do not match any Ingress path, unless the Ingress defines a `/` path or `spec.defaultBackend`. | *404* | | |*server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. Additionally, with the NGINX Plus, you can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | |*worker-processes* | Sets the value of the [worker_processes](https://nginx.org/en/docs/ngx_core_module.html#worker_processes) directive. | *auto* | | |*worker-rlimit-nofile* | Sets the value of the [worker_rlimit_nofile](https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile) directive. | N/A | | |*worker-connections* | Sets the value of the [worker_connections](https://nginx.org/en/docs/ngx_core_module.html#worker_connections) directive. | *1024* | | |*worker-cpu-affinity* | Sets the value of the [worker_cpu_affinity](https://nginx.org/en/docs/ngx_core_module.html#worker_cpu_affinity) directive. | N/A | | |*worker-shutdown-timeout* | Sets the value of the [worker_shutdown_timeout](https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout) directive. | N/A | | |*server-names-hash-bucket-size* | Sets the value of the [server_names_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size) directive. | *256* | | |*server-names-hash-max-size* | Sets the value of the [server_names_hash_max_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_max_size) directive. | *1024* | | |*map-hash-bucket-size* | Sets the value of the [map_hash_bucket_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size) directive.| *256* | | |*map-hash-max-size* | Sets the value of the [map_hash_max_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_max_size) directive. | *2048* | | |*resolver-addresses* | Sets the value of the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) addresses. Note: If you use a DNS name (for example, *kube-dns.kube-system.svc.cluster.local* ) as a resolver address, NGINX Plus will resolve it using the system resolver during the start and on every configuration reload. If the name cannot be resolved or the DNS server doesn't respond, NGINX Plus will fail to start or reload. To avoid this, we recommend using IP addresses as resolver addresses instead of DNS names. Supported in NGINX Plus only. | N/A | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*resolver-ipv6* | Enables IPv6 resolution in the resolver. Supported in NGINX Plus only. | *True* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*resolver-valid* | Sets the time NGINX caches the resolved DNS records. Supported in NGINX Plus only. | TTL value of a DNS record | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*resolver-timeout* | Sets the [resolver_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout) for name resolution. Supported in NGINX Plus only. | *30s* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*keepalive-timeout* | Sets the value of the [keepalive_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) directive. | *75s* | | |*keepalive-requests* | Sets the value of the [keepalive_requests](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests) directive. | *1000* | | |*variables-hash-bucket-size* | Sets the value of the [variables_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_bucket_size) directive. | *256* | | |*variables-hash-max-size* | Sets the value of the [variables-hash-max-size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_max_size) directive. | *1024* | | ### Logging |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*error-log-level* | Sets the global [error log level](https://nginx.org/en/docs/ngx_core_module.html#error_log) for NGINX. | *notice* | | |*access-log* | Sets the directive [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). A syslog destination is the only valid value. The value will be set to its default in-case user tries to configure it with location other than a syslog. | ``/dev/stdout main`` | ``syslog:server=localhost:514`` | |*access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). | *False* | | |*default-server-access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) for the default server. If access log is disabled globally (*access-log-off: "True"*), then the default server access log is always disabled. | *False* | | |*log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for HTTP and HTTPS traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v/internal/configs/version1/nginx.tmpl) for the access log. | [Custom Log Format](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/custom-log-format). | |*log-format-escaping* | Sets the characters escaping for the variables of the log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | |*stream-log-format* | Sets the custom [log format](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format) for TCP, UDP, and TLS Passthrough traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v/internal/configs/version1/nginx.tmpl). | | |*stream-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | ### Header manipulation **Note:** `disable-forwarded-headers` removes the hardcoded X-Forwarded-* headers directives allowing users to manually manage forwarded headers using add-header. This setting is available only when `--enable-snippets` is enabled, due to the security implications of manually managing forwarded headers. Only enable disable-forwarded-headers if: - You fully trust all upstream proxies in your request path, and - You have validated that your infrastructure sanitizes or overwrites these headers before they reach NGINX (for example, via an external load balancer), or you want to explicitly set these values yourself. |ConfigMap Key | Description | Default | | ---| ---| ---| |*proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: *"nginx.org/proxy-hide-headers": "header-a,header-b"* | N/A | |*proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: *"nginx.org/proxy-pass-headers": "header-a,header-b"* | N/A | |*add-header* | Adds one or more response headers with the [add_header](https://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header) directive in the `http` context. Use the format *Header-Name: value[:always]* and separate entries with commas. Example: `X-Frame-Options: DENY` or `X-Frame-Options: DENY: always`. | N/A | |*add-header-inherit* | Controls how [add_header_inherit](https://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header_inherit) applies inherited response headers. Allowed values are *on*, *off*, and *merge*. | N/A | |*disable-forwarded-headers* | Disables automatic injection of standard `X-Forwarded-*` headers (`X-Forwarded-For`, `X-Forwarded-Host`, `X-Forwarded-Port`, and `X-Forwarded-Proto`), allowing custom header management. Requires `-enable-snippets` CLI flag to be enabled.| *false* | ### Auth and SSL/TLS |ConfigMap Key | Description | Default | | ---| ---| ---| |*redirect-to-https* | Sets a redirect rule based on the value of the *http_x_forwarded_proto* header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of the Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115). The redirect code can be configured with the `http-redirect-code` key. | *False* | |*ssl-redirect* | Sets a redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS when TLS is configured. The redirect code can be configured with the `http-redirect-code` key. | *True* | |*http-redirect-code* | Sets the HTTP redirect code for HTTPS redirects. Supported codes: 301, 302, 307, 308. | *301* | |*hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/) : the HSTS header is added to the responses from backends. The *preload* directive is included in the header. Not applicable to VS/VSR. | *False* | |*hsts-max-age* | Sets the value of the *max-age* directive of the HSTS header. Not applicable to VS/VSR | *2592000* (1 month) | |*hsts-include-subdomains* | Adds the *includeSubDomains* directive to the HSTS header. Not applicable to VS/VSR | *False* | |*hsts-behind-proxy* | Enables HSTS based on the value of the *http_x_forwarded_proto* request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of the Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the *nginx.org/redirect-to-https* annotation. Not applicable to VS/VSR. | *False* | |*ssl-protocols* | Sets the value of the [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) directive. | *TLSv1 TLSv1.1 TLSv1.2* | |*ssl-prefer-server-ciphers* | Enables or disables the [ssl_prefer_server_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) directive. | *False* | |*ssl-ciphers* | Sets the value of the [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive. | *HIGH:!aNULL:!MD5* | |*ssl-dhparam-file* | Sets the content of the dhparam file. The controller will create the file and set the value of the [ssl_dhparam](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam) directive with the path of the file. | N/A | ### Listeners |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*http2* | Enables HTTP/2 in servers with SSL enabled. | *False* | | |*proxy-protocol* | Enables PROXY Protocol for incoming connections. | *False* | [Proxy Protocol](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/proxy-protocol). | ### Backend services (Upstreams) |ConfigMap Key | Description | Default | | ---| ---| ---| |*lb-method* | Sets the [load balancing method](/nginx/admin-guide/load-balancer/http-load-balancer.md#choosing-a-load-balancing-method). To use the round-robin method, specify *"round_robin"*. | *"random two least_conn"* | |*max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the *server* directive. | *1* | |*upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. For NGINX, the special value 0 disables the shared memory zones. For NGINX Plus, shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *256k* for NGINX, *512k* for NGINX Plus | |*fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the *server* directive. | *10s* | |*keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that *proxy_set_header Connection "";* is added to the generated configuration when the value > 0. | *0* | ### Zone Sync Zone Sync enables the [ngx_stream_zone_sync_module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) in NGINX Ingress Controller when NGINX Plus is used. Multiple replicas are required to effectively utililise this functionality. More information is available in the [How NGINX Plus Performs Zone Synchronization](/nginx/admin-guide/high-availability/zone_sync_details.md) topic. Zone synchronization with TLS for NGINX Ingress Controller is not yet available with ConfigMap. If you would like to enable Zone Sync with TLS, please remove `zone-sync` from ConfigMap and add Zone Sync parameters via [`stream-snippets`](/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md) similar to [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-config.yaml) and adding the [zone_sync_ssl directive](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl) along with any other TLS parameters to the `stream-snippets`. You will also need to manually add the headless service, such as in [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml). **Note:** If you previously installed OIDC or used the `zone_sync` directive with `stream-snippets` in [v4.0.1](https://github.com/nginx/kubernetes-ingress/tree/v4.0.1) or earlier, and you plan to enable the `zone-sync` ConfigMap key, the `zone_sync` directive should be removed from `stream-snippets`. If you encounter the error `error [emerg] 13#13: "zone_sync" directive is duplicate in /etc/nginx/nginx.conf:164` it is likely due to `zone_sync` being enabled in both `stream-snippets` and the ConfigMap. Once upgraded, remove the [old headless service](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml) deployed for OIDC. |ConfigMap Key | Description | Default | | ---| ---| ---| |*zone-sync* | Enables zone synchronization between NGINX Ingress Controller Pods. This autogenerates a [zone_sync_server](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_server) and a headless service using the `ReplicaSet`, `DaemonSet` or `StatefulSet` name. Please note that this headless service will be automatically cleaned up when uninstalling via Helm or by removing the value from the ConfigMap. The headless service will need to be manually removed if the `controller.customConfigMap` value is set via Helm or the deployment is uninstalled via Manifests. Each Ingress Controller manages its own headless service. NGINX Plus Required. | *False* | |*zone-sync-port* | Specifies the optional port on which NGINX Ingress Controller listens for zone sync traffic. NGINX Plus & `zone-sync` Required. | *12345* | |*zone-sync-resolver-addresses* | Configures optional addresses used in the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync. This field takes a comma separated list of addresses. NGINX Plus & `zone-sync` Required | `kube-dns.kube-system.svc.cluster.local` | |*zone-sync-resolver-ipv6* | Configures whether the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will look up IPv6 addresses. NGINX Plus & `zone-sync` Required | `true` | |*zone-sync-resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will override the TTL value of responses from nameservers with. NGINX Plus & `zone-sync` Required | `5s` | ### OIDC (OpenID Connect) Timeouts For more information on timeouts, see [here](https://github.com/nginxinc/nginx-openid-connect?tab=readme-ov-file#configuring-the-key-value-store) | ConfigMap Key | Description | Default | | ------------- | ------------| ------- | | *oidc-pkce-timeout* | Sets the timeout for PKCE (Proof Key for Code Exchange) in OIDC. | `90s` | | *oidc-id-tokens-timeout* | Sets the timeout for ID tokens in OIDC. | `1h` | | *oidc-access-tokens-timeout* | Sets the timeout for access tokens in OIDC. | `1h` | | *oidc-refresh-tokens-timeout* | Sets the timeout for refresh tokens in OIDC. | `8h` | | *oidc-sids-timeout* | Sets the timeout for session IDs in OIDC. | `8h` | ### OIDC (OpenID Connect) ZoneSizes For more information on zonesizes, see [here](https://github.com/nginxinc/nginx-openid-connect?tab=readme-ov-file#configuring-the-key-value-store) | ConfigMap Key | Description | Default | | ------------- | ------------| ------- | | *oidc-pkce-zone-size* | Sets the zonesize for PKCE (Proof Key for Code Exchange) in OIDC. | `128K` | | *oidc-id-tokens-zone-size* | Sets the zonesize for ID tokens in OIDC. | `1M` | | *oidc-access-tokens-zone-size* | Sets the zonesize for access tokens in OIDC. | `1M` | | *oidc-refresh-tokens-zone-size* | Sets the zonesize for refresh tokens in OIDC. | `1M` | | *oidc-sids-zone-size* | Sets the zonesize for session IDs in OIDC. | `1M` | ### Snippets and custom templates |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*main-snippets* | Sets a custom snippet in main context. | N/A | | |*http-snippets* | Sets a custom snippet in http context. | N/A | | |*location-snippets* | Sets a custom snippet in location context. | N/A | | |*server-snippets* | Sets a custom snippet in server context. | N/A | | |*stream-snippets* | Sets a custom snippet in stream context. | N/A | [Support for TCP/UDP Load Balancing](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/tcp-udp). | |*main-template* | Sets the main NGINX configuration template. | By default the template is read from the file in the container. | [Custom Templates](/nic/configuration/global-configuration/custom-templates.md). | |*ingress-template* | Sets the NGINX configuration template for an Ingress resource. | By default the template is read from the file on the container. | [Custom Templates](/nic/configuration/global-configuration/custom-templates.md). | |*virtualserver-template* | Sets the NGINX configuration template for an VirtualServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nic/configuration/global-configuration/custom-templates.md). | |*transportserver-template* | Sets the NGINX configuration template for a TransportServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nic/configuration/global-configuration/custom-templates.md) | ### Modules |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*otel-exporter-endpoint* | OTLP/gRPC endpoint that will accept [OpenTelemetry](https://opentelemetry.io) data. Set `otel-trace-in-http` to *"true"* to enable OpenTelemetry at the global level. | N/A | *"https://otel-collector:4317"* | |*otel-exporter-header-name* | The name of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-value` required. | N/A | *"X-custom-header"* | |*otel-exporter-header-value* | The value of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-name` required. | N/A | *"custom-value"* | |*otel-service-name* | Sets the `service.name` attribute of the OTel resource. `otel-exporter-endpoint` required. | N/A | *"nginx-ingress-controller:nginx"* | | *otel-trace-in-http* | Enables [OpenTelemetry](https://opentelemetry.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Set this to *"false"* to enable OpenTelemetry for individual routes with snippets. `otel-exporter-endpoint` required. | *"false"* | *"true"* | |*opentracing* | Removed in v5.0.0. Enables [OpenTracing](https://opentracing.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Note: requires the Ingress Controller image with OpenTracing module and a tracer. See the [docs](/nic/logging-and-monitoring/opentracing.md) for more information. | *False* | | |*opentracing-tracer* | Removed in v5.0.0. Sets the path to the vendor tracer binary plugin. | N/A | | |*opentracing-tracer-config* | Removed in v5.0.0. Sets the tracer configuration in JSON format. | N/A | | |*app-protect-compressed-requests-action* | Sets the *app_protect_compressed_requests_action* [global directive](/nginx-app-protect/configuration/#global-directives). | *drop* | | |*app-protect-cookie-seed* | Sets the *app_protect_cookie_seed* [global directive](/nginx-app-protect/configuration/#global-directives). | Random automatically generated string | | |*app-protect-failure-mode-action* | Sets the *app_protect_failure_mode_action* [global directive](/nginx-app-protect/configuration/#global-directives). | *pass* | | |*app-protect-cpu-thresholds* | Sets the *app_protect_cpu_thresholds* [global directive](/nginx-app-protect/configuration/#global-directives). | *high=100 low=100* | | |*app-protect-physical-memory-util-thresholds* | Sets the *app_protect_physical_memory_util_thresholds* [global directive](/nginx-app-protect/configuration/#global-directives). | *high=100 low=100* | | |`app-protect-reconnect-period-seconds` | Sets the `app_protect_reconnect_period_seconds` [global directive](/nginx-app-protect/configuration/#global-directives). | `5` | | |*app-protect-dos-log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for Dos Access log traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | `, vs_name_al=$app_protect_dos_vs_name, ip=$remote_addr, tls_fp=$app_protect_dos_tls_fp, outcome=$app_protect_dos_outcome, reason=$app_protect_dos_outcome_reason, policy_name=$app_protect_dos_policy_name, dos_version=$app_protect_dos_version, ip_tls=$remote_addr:$app_protect_dos_tls_fp,` | | |*app-protect-dos-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | |*app-protect-dos-arb-fqdn* | Sets the *app-protect-dos-arb-fqdn* [directive](/nginx-app-protect-dos/directives-and-policy/learn-about-directives-and-policy/#arbitrator-fqdn-directive-app_protect_dos_arb_fqdn). | *svc-appprotect-dos-arb* | | -------------------------------------------------------------------------------- title: "ConfigMap resources" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/global-configuration/configmap-resource/index.md -------------------------------------------------------------------------------- When using F5 NGINX Ingress Controller LTS, you can customize or fine tune NGINX behavior using ConfigMap resources. Examples include setting the number of worker processes or customizing the access log format. ## Using ConfigMap 1. The [Installation with Manifests](/nic/lts/install/manifests.md) documentation deploy an empty ConfigMap while the default installation manifests specify it in the command-line arguments of the Ingress Controller. However, if you customized the manifests, to use ConfigMap, make sure to specify the ConfigMap resource to use the [command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments) of NGINX Ingress Controller LTS. 1. Create a ConfigMap file with the name *nginx-config.yaml* and set the values that make sense for your setup: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-config namespace: nginx-ingress data: proxy-connect-timeout: "10s" proxy-read-timeout: "10s" client-max-body-size: "2m" ``` See the section [Summary of ConfigMap Keys](#configmap-keys) for the explanation of the available ConfigMap keys (such as `proxy-connect-timeout` in this example). 1. Create a new (or update the existing) ConfigMap resource: ```shell kubectl apply -f nginx-config.yaml ``` The NGINX configuration will be updated. --- ## ConfigMap and Ingress annotations ConfigMap applies globally, meaning that it affects every Ingress resource. In contrast, annotations always apply to their Ingress resource. Annotations can override some ConfigMap keys: an example is that the `nginx.org/proxy-connect-timeout` annotations overrides the `proxy-connect-timeout` ConfigMap key. For more information, view the [Advanced configuration with annotations](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations) topic. ## ConfigMap and VirtualServer/VirtualServerRoute resources The ConfigMap affects every VirtualServer and VirtualServerRoute resources. However, the fields of those resources allow overriding some ConfigMap keys. For example, the `connect-timeout` field of the `upstream` overrides the `proxy-connect-timeout` ConfigMap key. For more information, view the [VirtualServer and VirtualServerRoute resources](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources) topic. ## ConfigMap keys ### Ingress Controller (Unrelated to NGINX Configuration) |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*external-status-address* | Sets the address to be reported in the status of Ingress resources. Requires the *-report-status* command-line argument. Overrides the *-external-service* argument. | N/A | [Reporting resource status](/nic/lts/configuration/global-configuration/reporting-resources-status) | ### General customization |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*proxy-connect-timeout* | Sets the value of the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) and [grpc_connect_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_connect_timeout) directive. | *60s* | | |*proxy-read-timeout* | Sets the value of the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) and [grpc_read_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_read_timeout) directive. | *60s* | | |*proxy-send-timeout* | Sets the value of the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) and [grpc_send_timeout](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_send_timeout) directive. | *60s* | | |*client-max-body-size* | Sets the value of the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. | *1m* | | |*client-body-buffer-size* | Sets the value of the [client_body_buffer_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) directive. | N/A | | |*proxy-buffering* | Enables or disables [buffering of responses](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) from the proxied server. | *True* | | |*proxy-buffers* | Sets the value of the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive. | Depends on the platform. | | |*proxy-buffer-size* | Sets the value of the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) and [grpc_buffer_size](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_buffer_size) directives. | Depends on the platform. | | |*proxy-busy-buffers-size* | Sets the value of the [proxy_busy_buffers_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) directive. | Depends on the platform. | | |*proxy-max-temp-file-size* | Sets the value of the [proxy_max_temp_file_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size) directive. | *1024m* | | |*set-real-ip-from* | Sets the value of the [set_real_ip_from](https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from) directive. | N/A | | |*real-ip-header* | Sets the value of the [real_ip_header](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header) directive. | *X-Real-IP* | | |*real-ip-recursive* | Enables or disables the [real_ip_recursive](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive) directive. | *False* | | |*default-server-return* | Configures the [return](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return) directive in the default server, which handles a client request if none of the hosts of Ingress or VirtualServer resources match. The default value configures NGINX to return a 404 error page. You can configure a fixed response or a redirect. For example, *default-server-return: 302 https://nginx.org* will redirect a client to *https://nginx.org*. | *404* | | |*server-tokens* | Enables or disables the [server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) directive. You can specify a custom string value, including the empty string value, which disables the emission of the “Server” field. | *True* | | |*worker-processes* | Sets the value of the [worker_processes](https://nginx.org/en/docs/ngx_core_module.html#worker_processes) directive. | *auto* | | |*worker-rlimit-nofile* | Sets the value of the [worker_rlimit_nofile](https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile) directive. | N/A | | |*worker-connections* | Sets the value of the [worker_connections](https://nginx.org/en/docs/ngx_core_module.html#worker_connections) directive. | *1024* | | |*worker-cpu-affinity* | Sets the value of the [worker_cpu_affinity](https://nginx.org/en/docs/ngx_core_module.html#worker_cpu_affinity) directive. | N/A | | |*worker-shutdown-timeout* | Sets the value of the [worker_shutdown_timeout](https://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout) directive. | N/A | | |*server-names-hash-bucket-size* | Sets the value of the [server_names_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size) directive. | *256* | | |*server-names-hash-max-size* | Sets the value of the [server_names_hash_max_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_max_size) directive. | *1024* | | |*map-hash-bucket-size* | Sets the value of the [map_hash_bucket_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size) directive.| *256* | | |*map-hash-max-size* | Sets the value of the [map_hash_max_size](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_max_size) directive. | *2048* | | |*resolver-addresses* | Sets the value of the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) addresses. Note: If you use a DNS name (for example, *kube-dns.kube-system.svc.cluster.local* ) as a resolver address, NGINX Plus will resolve it using the system resolver during the start and on every configuration reload. If the name cannot be resolved or the DNS server doesn't respond, NGINX Plus will fail to start or reload. To avoid this, we recommend using IP addresses as resolver addresses instead of DNS names. | N/A | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*resolver-ipv6* | Enables IPv6 resolution in the resolver. | *True* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*resolver-valid* | Sets the time NGINX caches the resolved DNS records. | TTL value of a DNS record | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*resolver-timeout* | Sets the [resolver_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout) for name resolution. | *30s* | [Support for Type ExternalName Services](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services). | |*keepalive-timeout* | Sets the value of the [keepalive_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) directive. | *75s* | | |*keepalive-requests* | Sets the value of the [keepalive_requests](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests) directive. | *1000* | | |*variables-hash-bucket-size* | Sets the value of the [variables_hash_bucket_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_bucket_size) directive. | *256* | | |*variables-hash-max-size* | Sets the value of the [variables-hash-max-size](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_max_size) directive. | *1024* | | ### Logging |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*error-log-level* | Sets the global [error log level](https://nginx.org/en/docs/ngx_core_module.html#error_log) for NGINX. | *notice* | | |*access-log* | Sets the directive [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). A syslog destination is the only valid value. The value will be set to its default in-case user tries to configure it with location other than a syslog. | ``/dev/stdout main`` | ``syslog:server=localhost:514`` | |*access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log). | *False* | | |*default-server-access-log-off* | Disables the [access log](https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) for the default server. If access log is disabled globally (*access-log-off: "True"*), then the default server access log is always disabled. | *False* | | |*log-format* | Sets the custom [log format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for HTTP and HTTPS traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v/internal/configs/version1/nginx.tmpl) for the access log. | [Custom Log Format](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/custom-log-format). | |*log-format-escaping* | Sets the characters escaping for the variables of the log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | |*stream-log-format* | Sets the custom [log format](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format) for TCP, UDP, and TLS Passthrough traffic. For convenience, it is possible to define the log format across multiple lines (each line separated by *\n*). In that case, the Ingress Controller will replace every *\n* character with a space character. All *'* characters must be escaped. | See the [template file](https://github.com/nginx/kubernetes-ingress/blob/v/internal/configs/version1/nginx.tmpl). | | |*stream-log-format-escaping* | Sets the characters escaping for the variables of the stream log format. Supported values: *json* (JSON escaping), *default* (the default escaping) *none* (disables escaping). | *default* | | ### Request URI/Header manipulation |ConfigMap Key | Description | Default | | ---| ---| ---| |*proxy-hide-headers* | Sets the value of one or more [proxy_hide_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directives. Example: *"nginx.org/proxy-hide-headers": "header-a,header-b"* | N/A | |*proxy-pass-headers* | Sets the value of one or more [proxy_pass_header](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directives. Example: *"nginx.org/proxy-pass-headers": "header-a,header-b"* | N/A | ### Auth and SSL/TLS |ConfigMap Key | Description | Default | | ---| ---| ---| |*redirect-to-https* | Sets a redirect rule based on the value of the *http_x_forwarded_proto* header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of the Ingress Controller — see [115](https://github.com/nginx/kubernetes-ingress/issues/115). The redirect code can be configured with the `http-redirect-code` key. | *False* | |*ssl-redirect* | Sets a redirect rule for all incoming HTTP traffic to force incoming traffic over HTTPS when TLS is configured. The redirect code can be configured with the `http-redirect-code` key. | *True* | |*http-redirect-code* | Sets the HTTP redirect code for HTTPS redirects. Supported codes: 301, 302, 307, 308. | *301* | |*hsts* | Enables [HTTP Strict Transport Security (HSTS)](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/) : the HSTS header is added to the responses from backends. The *preload* directive is included in the header. | *False* | |*hsts-max-age* | Sets the value of the *max-age* directive of the HSTS header. | *2592000* (1 month) | |*hsts-include-subdomains* | Adds the *includeSubDomains* directive to the HSTS header. | *False* | |*hsts-behind-proxy* | Enables HSTS based on the value of the *http_x_forwarded_proto* request header. Should only be used when TLS termination is configured in a load balancer (proxy) in front of the Ingress Controller. Note: to control redirection from HTTP to HTTPS configure the *nginx.org/redirect-to-https* annotation. | *False* | |*ssl-protocols* | Sets the value of the [ssl_protocols](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) directive. | *TLSv1 TLSv1.1 TLSv1.2* | |*ssl-prefer-server-ciphers* | Enables or disables the [ssl_prefer_server_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) directive. | *False* | |*ssl-ciphers* | Sets the value of the [ssl_ciphers](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) directive. | *HIGH:!aNULL:!MD5* | |*ssl-dhparam-file* | Sets the content of the dhparam file. The controller will create the file and set the value of the [ssl_dhparam](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam) directive with the path of the file. | N/A | ### Listeners |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*http2* | Enables HTTP/2 in servers with SSL enabled. | *False* | | |*proxy-protocol* | Enables PROXY Protocol for incoming connections. | *False* | [Proxy Protocol](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/proxy-protocol). | ### Backend services (Upstreams) |ConfigMap Key | Description | Default | | ---| ---| ---| |*lb-method* | Sets the [load balancing method](/nginx/admin-guide/load-balancer/http-load-balancer.md#choosing-a-load-balancing-method). To use the round-robin method, specify *"round_robin"*. | *"random two least_conn"* | |*max-fails* | Sets the value of the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the *server* directive. | *1* | |*upstream-zone-size* | Sets the size of the shared memory [zone](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#zone) for upstreams. Shared memory zones are required and cannot be disabled. The special value 0 will be ignored. | *512k* for NGINX Plus | |*fail-timeout* | Sets the value of the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the *server* directive. | *10s* | |*keepalive* | Sets the value of the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. Note that *proxy_set_header Connection "";* is added to the generated configuration when the value > 0. | *0* | ### Zone Sync Zone Sync enables the [ngx_stream_zone_sync_module](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html) in NGINX Ingress Controller LTS. Multiple replicas are required to effectively utililise this functionality. More information is available in the [How NGINX Plus Performs Zone Synchronization](/nginx/admin-guide/high-availability/zone_sync_details.md) topic. Zone synchronization with TLS for NGINX Ingress Controller LTS is not yet available with ConfigMap. If you would like to enable Zone Sync with TLS, please remove `zone-sync` from ConfigMap and add Zone Sync parameters via [`stream-snippets`](/nic/lts/configuration/ingress-resources/advanced-configuration-with-snippets.md) similar to [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-config.yaml) and adding the [zone_sync_ssl directive](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_ssl) along with any other TLS parameters to the `stream-snippets`. You will also need to manually add the headless service, such as in [this example](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml). **caution:** If you previously installed OIDC or used the `zone_sync` directive with `stream-snippets` in [v4.0.1](https://github.com/nginx/kubernetes-ingress/tree/v4.0.1) or earlier, and you plan to enable the `zone-sync` ConfigMap key, the `zone_sync` directive should be removed from `stream-snippets`. If you encounter the error `error [emerg] 13#13: "zone_sync" directive is duplicate in /etc/nginx/nginx.conf:164` it is likely due to `zone_sync` being enabled in both `stream-snippets` and the ConfigMap. Once upgraded, remove the [old headless service](https://github.com/nginx/kubernetes-ingress/blob/v4.0.1/examples/custom-resources/oidc/nginx-ingress-headless.yaml) deployed for OIDC. |ConfigMap Key | Description | Default | | ---| ---| ---| |*zone-sync* | Enables zone synchronization between NGINX Ingress Controller LTS Pods. This autogenerates a [zone_sync_server](https://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html#zone_sync_server) and a headless service using the `ReplicaSet`, `DaemonSet` or `StatefulSet` name. Please note that this headless service will be automatically cleaned up when uninstalling via Helm or by removing the value from the ConfigMap. The headless service will need to be manually removed if the `controller.customConfigMap` value is set via Helm or the deployment is uninstalled via Manifests. Each Ingress Controller manages its own headless service. | *False* | |*zone-sync-port* | Specifies the optional port on which NGINX Ingress Controller LTS listens for zone sync traffic. `zone-sync` Required. | *12345* | |*zone-sync-resolver-addresses* | Configures optional addresses used in the [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync. This field takes a comma separated list of addresses. `zone-sync` Required | `kube-dns.kube-system.svc.cluster.local` | |*zone-sync-resolver-ipv6* | Configures whether the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will look up IPv6 addresses. `zone-sync` Required | `true` | |*zone-sync-resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the optional [resolver](https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver) directive for zone-sync will override the TTL value of responses from nameservers with. `zone-sync` Required | `5s` | ### OIDC (OpenID Connect) Timeouts For more information on timeouts, see [here](https://github.com/nginxinc/nginx-openid-connect?tab=readme-ov-file#configuring-the-key-value-store) | ConfigMap Key | Description | Default | | ------------- | ------------| ------- | | *oidc-pkce-timeout* | Sets the timeout for PKCE (Proof Key for Code Exchange) in OIDC. | `90s` | | *oidc-id-tokens-timeout* | Sets the timeout for ID tokens in OIDC. | `1h` | | *oidc-access-tokens-timeout* | Sets the timeout for access tokens in OIDC. | `1h` | | *oidc-refresh-tokens-timeout* | Sets the timeout for refresh tokens in OIDC. | `24h` | | *oidc-sids-timeout* | Sets the timeout for session IDs in OIDC. | `24h` | ### Snippets and custom templates |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*main-snippets* | Sets a custom snippet in main context. | N/A | | |*http-snippets* | Sets a custom snippet in http context. | N/A | | |*location-snippets* | Sets a custom snippet in location context. | N/A | | |*server-snippets* | Sets a custom snippet in server context. | N/A | | |*stream-snippets* | Sets a custom snippet in stream context. | N/A | [Support for TCP/UDP Load Balancing](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/tcp-udp). | |*main-template* | Sets the main NGINX configuration template. | By default the template is read from the file in the container. | [Custom Templates](/nic/lts/configuration/global-configuration/custom-templates.md). | |*ingress-template* | Sets the NGINX configuration template for an Ingress resource. | By default the template is read from the file on the container. | [Custom Templates](/nic/lts/configuration/global-configuration/custom-templates.md). | |*virtualserver-template* | Sets the NGINX configuration template for an VirtualServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nic/lts/configuration/global-configuration/custom-templates.md). | |*transportserver-template* | Sets the NGINX configuration template for a TransportServer resource. | By default the template is read from the file on the container. | [Custom Templates](/nic/lts/configuration/global-configuration/custom-templates.md) | ### Modules |ConfigMap Key | Description | Default | Example | | ---| ---| ---| --- | |*otel-exporter-endpoint* | OTLP/gRPC endpoint that will accept [OpenTelemetry](https://opentelemetry.io) data. Set `otel-trace-in-http` to *"true"* to enable OpenTelemetry at the global level. | N/A | *"https://otel-collector:4317"* | |*otel-exporter-header-name* | The name of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-value` required. | N/A | *"X-custom-header"* | |*otel-exporter-header-value* | The value of a custom HTTP header to add to telemetry export request. `otel-exporter-endpoint` and `otel-exporter-header-name` required. | N/A | *"custom-value"* | |*otel-service-name* | Sets the `service.name` attribute of the OTel resource. `otel-exporter-endpoint` required. | N/A | *"nginx-ingress-controller:nginx"* | |*otel-trace-in-http* | Enables [OpenTelemetry](https://opentelemetry.io) globally (for all Ingress, VirtualServer and VirtualServerRoute resources). Set this to *"false"* to enable OpenTelemetry for individual routes with snippets. `otel-exporter-endpoint` required. | *"false"* | *"true"* | -------------------------------------------------------------------------------- title: "Create a license Secret" url: https://docs.nginx.com/nginx-ingress-controller/install/license-secret/index.md -------------------------------------------------------------------------------- This document explains how to create and use a license secret for F5 NGINX Ingress Controller. # Overview NGINX Plus Ingress Controller requires a valid JSON Web Token (JWT) to download the container image from the F5 registry. From version 4.0.0, this JWT token is also required to run NGINX Plus. This requirement is part of F5’s broader licensing program and aligns with industry best practices. The JWT will streamline subscription renewals and usage reporting, helping you manage your NGINX Plus subscription more efficiently. The [telemetry](#telemetry) data we collect helps us improve our products and services to better meet your needs. The JWT is required for validating your subscription and reporting telemetry data. For environments connected to the internet, telemetry is automatically sent to F5’s licensing endpoint. In offline environments, telemetry is routed through [NGINX Instance Manager](/nim/). By default usage is reported every hour and also whenever NGINX is reloaded. **Note:** Read the [subscription licenses topic](/solutions/about-subscription-licenses.md#for-internet-connected-environments) for a list of IPs associated with F5's licensing endpoint (`product.connect.nginx.com`). ## Set up your NGINX Plus license ### Download the JWT 1. Log in to [MyF5](https://my.f5.com/manage/s/). 2. Go to **My Products & Plans > Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription, and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. A regular NGINX Plus instance JWT must be used. ### Create the Secret The JWT needs to be configured before deploying NGINX Ingress Controller. It must be stored in a Kubernetes Secret of type `nginx.com/license` in the same namespace as your NGINX Ingress Controller pod(s). Create the Secret with the following command: ```shell kubectl create secret generic license-token --from-file=license.jwt= --type=nginx.com/license -n ``` Once created, you can download the `.jwt` file. **Note:** For security, follow these practices with JSON Web Tokens (JWTs), passwords, and shell history: 1. **JWTs:** JWTs are sensitive information. Store them securely. Delete them after use to prevent unauthorized access. 1. **Shell history:** Commands that include JWTs or passwords are recorded in the history of your shell, in plain text. Clear your shell history after running such commands. For example, if you use bash, you can delete commands in your `~/.bash_history` file. Alternatively, you can run the `history -c` command to erase your shell history. Follow these practices to help ensure the security of your system and data. ### Update the Secret If you've already deployed NGINX Ingress Controller and need to rotate or renew the JWT (for example, when the existing token is about to expire or has been replaced), update the existing Secret in place. First, take your new JWT license token, and save it to your existing `license.jwt` file. Next, use the following command to generate the updated Secret manifest and apply it: ```shell kubectl create secret generic license-token \ --save-config \ --dry-run=client \ --from-file=license.jwt= \ --type=nginx.com/license \ -o yaml | \ kubectl apply -f - ``` Notes: - Replace `license.jwt` on the `--from-file` flag with the path to your renewed JWT file if it's not in the current directory. - If your Secret resides in a specific namespace, include `-n ` on the `kubectl create secret` command so the generated YAML contains the correct namespace. - Ensure the Secret name (`license-token` by default) matches the name referenced by your Helm values or Management ConfigMap. - After the Secret is updated, the mounted Secret volume in the Pod is refreshed automatically by Kubernetes. NGINX Plus applies the updated license automatically. If you do not see the update take effect after a short period, restart the Ingress Controller Pod(s) to force a re-read of the Secret. ### Add the license Secret to your deployment If using a name other than the default `license-token`, provide the name of this Secret when installing NGINX Ingress Controller: #### Helm Specify the Secret name using the `controller.mgmt.licenseTokenSecretName` Helm value. For detailed guidance on creating the Management block with Helm, refer to the [Helm installation topics](/nic/install/helm/). #### Manifests Specify the Secret name in the `license-token-secret-name` Management ConfigMap key. For detailed guidance on creating the Management ConfigMap, refer to the [Management ConfigMap Resource Documentation](/nic/configuration/global-configuration/mgmt-configmap-resource/). If you are reporting to the default licensing endpoint, then you can now proceed with [installing NGINX Ingress Controller](/nic/install/). Otherwise, follow the steps below to configure reporting to NGINX Instance Manager ### Create report for NGINX Instance Manager {#nim} If you are deploying NGINX Ingress Controller in an "air-gapped" environment you will need to report to [NGINX Instance Manager](/nim/) instead of the default licensing endpoint. First, you must specify the endpoint of your NGINX Instance Manager. #### Helm Specify the endpoint using the `controller.mgmt.usageReport.endpoint` helm value. #### Manifests Specify the endpoint in the `usage-report-endpoint` Management ConfigMap key. #### Configure SSL certificates and SSL trusted certificates {#nim-cert} To configure SSL certificates or SSL trusted certificates, extra steps are necessary. To use Client Auth with NGINX Instance Manager, first create a Secret of type `kubernetes.io/tls` in the same namespace as the NGINX Ingress Controller pods. ```shell kubectl create secret tls ssl-certificate --cert= --key= -n ``` To provide a SSL trusted certificate, and an optional Certificate Revocation List, create a Secret of type `nginx.org/ca` in the Namespace that the NIC Pod(s) are in. ```shell kubectl create secret generic ssl-trusted-certificate \ --from-file=ca.crt= \ --from-file=ca.crl= \ # optional --type=nginx.org/ca ``` Providing an optional CRL (certificate revocation list) will configure the [`ssl_crl`](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directive. #### Helm Specify the SSL certificate Secret name using the `controller.mgmt.sslCertificateSecretName` Helm value. Specify the SSL trusted certificate Secret name using the `controller.mgmt.sslTrustedCertificateSecretName` Helm value. #### Manifests Specify the SSL certificate Secret name in the `ssl-certificate-secret-name` management ConfigMap key. Specify the SSL trusted certificate Secret name in the `ssl-trusted-certificate-secret-name` management ConfigMap key. Once these Secrets are created and configured, you can now [install NGINX Ingress Controller ](/nic/install/). ## What’s reported and how it’s protected {#telemetry} NGINX Plus reports the following data every hour by default: - **NGINX version and status**: The version of NGINX Plus running on the instance. - **Instance UUID**: A unique identifier for each NGINX Plus instance. - **Traffic data**: - **Bytes received from and sent to clients**: HTTP and stream traffic volume between clients and NGINX Plus. - **Bytes received from and sent to upstreams**: HTTP and stream traffic volume between NGINX Plus and upstream servers. - **Client connections**: The number of accepted client connections (HTTP and stream traffic). - **Requests handled**: The total number of HTTP requests processed. - **NGINX uptime**: The number of reloads and worker connections during uptime. - **Usage report timestamps**: Start and end times for each usage report. - **Kubernetes node details**: Information about Kubernetes nodes. ### Security and privacy of reported data All communication between your NGINX Plus instances, NGINX Instance Manager, and F5’s licensing endpoint (`product.connect.nginx.com`) is protected using **SSL/TLS** encryption. Only **operational metrics** are reported — no **personally identifiable information (PII)** or **sensitive customer data** is transmitted. -------------------------------------------------------------------------------- title: "Create a license Secret" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/license-secret/index.md -------------------------------------------------------------------------------- This document explains how to create and use a license secret for F5 NGINX Ingress Controller LTS. # Overview NGINX Plus Ingress Controller requires a valid JSON Web Token (JWT) to download the container image from the F5 registry. This JWT token is also required to run NGINX Plus. This requirement is part of F5’s broader licensing program and aligns with industry best practices. The JWT will streamline subscription renewals and usage reporting, helping you manage your NGINX Plus subscription more efficiently. The [telemetry](#telemetry) data we collect helps us improve our products and services to better meet your needs. The JWT is required for validating your subscription and reporting telemetry data. For environments connected to the internet, telemetry is automatically sent to F5’s licensing endpoint. In offline environments, telemetry is routed through [NGINX Instance Manager](/nim/). By default usage is reported every hour and also whenever NGINX is reloaded. **note:** Read the [subscription licenses topic](/solutions/about-subscription-licenses.md#for-internet-connected-environments) for a list of IPs associated with F5's licensing endpoint (`product.connect.nginx.com`). ## Set up your NGINX Plus license ### Download the JWT 1. Log in to [MyF5](https://my.f5.com/manage/s/). 2. Go to **My Products & Plans > Subscriptions** to see your active subscriptions. 3. Find your NGINX products or services subscription, and select the **Subscription ID** for details. 4. Download the **JSON Web Token (JWT)** from the subscription page. **Note:** The Connectivity Stack for Kubernetes JWT does not work with NGINX Plus reporting. A regular NGINX Plus instance JWT must be used. ### Create the Secret The JWT needs to be configured before deploying NGINX Ingress Controller LTS. It must be stored in a Kubernetes Secret of type `nginx.com/license` in the same namespace as your NGINX Ingress Controller LTS pod(s). Create the Secret with the following command: ```shell kubectl create secret generic license-token --from-file=license.jwt= --type=nginx.com/license -n ``` Once created, you can download the `.jwt` file. **Note:** For security, follow these practices with JSON Web Tokens (JWTs), passwords, and shell history: 1. **JWTs:** JWTs are sensitive information. Store them securely. Delete them after use to prevent unauthorized access. 1. **Shell history:** Commands that include JWTs or passwords are recorded in the history of your shell, in plain text. Clear your shell history after running such commands. For example, if you use bash, you can delete commands in your `~/.bash_history` file. Alternatively, you can run the `history -c` command to erase your shell history. Follow these practices to help ensure the security of your system and data. ### Update the Secret If you've already deployed NGINX Ingress Controller LTS and need to rotate or renew the JWT (for example, when the existing token is about to expire or has been replaced), update the existing Secret in place. First, take your new JWT license token, and save it to your existing `license.jwt` file. Next, use the following command to generate the updated Secret manifest and apply it: ```shell kubectl create secret generic license-token \ --save-config \ --dry-run=client \ --from-file=license.jwt= \ --type=nginx.com/license \ -o yaml | \ kubectl apply -f - ``` Notes: - Replace `license.jwt` on the `--from-file` flag with the path to your renewed JWT file if it's not in the current directory. - If your Secret resides in a specific namespace, include `-n ` on the `kubectl create secret` command so the generated YAML contains the correct namespace. - Ensure the Secret name (`license-token` by default) matches the name referenced by your Helm values or Management ConfigMap. - After the Secret is updated, the mounted Secret volume in the Pod is refreshed automatically by Kubernetes. NGINX Plus applies the updated license automatically. If you do not see the update take effect after a short period, restart the Ingress Controller Pod(s) to force a re-read of the Secret. ### Add the license Secret to your deployment If using a name other than the default `license-token`, provide the name of this Secret when installing NGINX Ingress Controller LTS: #### Helm Specify the Secret name using the `controller.mgmt.licenseTokenSecretName` Helm value. For detailed guidance on creating the Management block with Helm, refer to the [Helm installation topics](/nic/lts/install/helm/). #### Manifests Specify the Secret name in the `license-token-secret-name` Management ConfigMap key. For detailed guidance on creating the Management ConfigMap, refer to the [Management ConfigMap Resource Documentation](/nic/lts/configuration/global-configuration/mgmt-configmap-resource/). If you are reporting to the default licensing endpoint, then you can now proceed with [installing NGINX Ingress Controller LTS](/nic/lts/install/). Otherwise, follow the steps below to configure reporting to NGINX Instance Manager ### Create report for NGINX Instance Manager {#nim} If you are deploying NGINX Ingress Controller LTS in an "air-gapped" environment you will need to report to [NGINX Instance Manager](/nim/) instead of the default licensing endpoint. First, you must specify the endpoint of your NGINX Instance Manager. #### Helm Specify the endpoint using the `controller.mgmt.usageReport.endpoint` helm value. #### Manifests Specify the endpoint in the `usage-report-endpoint` Management ConfigMap key. #### Configure SSL certificates and SSL trusted certificates {#nim-cert} To configure SSL certificates or SSL trusted certificates, extra steps are necessary. To use Client Auth with NGINX Instance Manager, first create a Secret of type `kubernetes.io/tls` in the same namespace as the NGINX Ingress Controller LTS pods. ```shell kubectl create secret tls ssl-certificate --cert= --key= -n ``` To provide a SSL trusted certificate, and an optional Certificate Revocation List, create a Secret of type `nginx.org/ca` in the Namespace that the NIC Pod(s) are in. ```shell kubectl create secret generic ssl-trusted-certificate \ --from-file=ca.crt= \ --from-file=ca.crl= \ # optional --type=nginx.org/ca ``` Providing an optional CRL (certificate revocation list) will configure the [`ssl_crl`](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directive. #### Helm Specify the SSL certificate Secret name using the `controller.mgmt.sslCertificateSecretName` Helm value. Specify the SSL trusted certificate Secret name using the `controller.mgmt.sslTrustedCertificateSecretName` Helm value. #### Manifests Specify the SSL certificate Secret name in the `ssl-certificate-secret-name` management ConfigMap key. Specify the SSL trusted certificate Secret name in the `ssl-trusted-certificate-secret-name` management ConfigMap key. Once these Secrets are created and configured, you can now [install NGINX Ingress Controller LTS](/nic/lts/install/). ## What’s reported and how it’s protected {#telemetry} NGINX Plus reports the following data every hour by default: - **NGINX version and status**: The version of NGINX Plus running on the instance. - **Instance UUID**: A unique identifier for each NGINX Plus instance. - **Traffic data**: - **Bytes received from and sent to clients**: HTTP and stream traffic volume between clients and NGINX Plus. - **Bytes received from and sent to upstreams**: HTTP and stream traffic volume between NGINX Plus and upstream servers. - **Client connections**: The number of accepted client connections (HTTP and stream traffic). - **Requests handled**: The total number of HTTP requests processed. - **NGINX uptime**: The number of reloads and worker connections during uptime. - **Usage report timestamps**: Start and end times for each usage report. - **Kubernetes node details**: Information about Kubernetes nodes. ### Security and privacy of reported data All communication between your NGINX Plus instances, NGINX Instance Manager, and F5’s licensing endpoint (`product.connect.nginx.com`) is protected using **SSL/TLS** encryption. Only **operational metrics** are reported — no **personally identifiable information (PII)** or **sensitive customer data** is transmitted. -------------------------------------------------------------------------------- title: "Custom annotations" url: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/custom-annotations/index.md -------------------------------------------------------------------------------- This topic explains how you can use custom annotations with F5 NGINX Ingress Controller. Custom annotations enable you to quickly extend the Ingress resource to support many advanced features of NGINX, such as rate limiting, caching, etc. ## Overview NGINX Ingress Controller supports a number of annotations for the Ingress resource that fine tune NGINX configuration (for example, connection timeouts) or enable additional features (for example, JWT validation). The complete list of annotations is available [here](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md). The annotations are provided only for the most common features and use cases, meaning that not every NGINX feature or a customization option is available through the annotations. Additionally, even if an annotation is available, it might not give you the satisfactory level of control of a particular NGINX feature. Custom annotations allow you to add an annotation for an NGINX feature that is not available as a regular annotation. In contrast with regular annotations, to add a custom annotation, you don't need to modify the Ingress Controller source code -- just modify the template. Additionally, with a custom annotation, you get full control of how the feature is implemented in NGINX configuration. ## Usage The Ingress Controller generates NGINX configuration for Ingress resources by executing a configuration template. See [NGINX template](https://github.com/nginx/kubernetes-ingress/blob/v/internal/configs/version1/nginx.ingress.tmpl) or [NGINX Plus template](https://github.com/nginx/kubernetes-ingress/blob/v/internal/configs/version1/nginx-plus.ingress.tmpl). To support custom annotations, the template has access to the information about the Ingress resource - its *name*, *namespace* and *annotations*. It is possible to check if a particular annotation present in the Ingress resource and conditionally insert NGINX configuration directives at multiple NGINX contexts - `http`, `server`, `location` or `upstream`. Additionally, you can get the value that is set to the annotation. Consider the following excerpt from the template, which was extended to support two custom annotations: ```jinja2 # This is the configuration for {{$.Ingress.Name}}/{{$.Ingress.Namespace}} {{if index $.Ingress.Annotations "custom.nginx.org/feature-a"}} # Insert config for feature A if the annotation is set {{end}} {{with $value := index $.Ingress.Annotations "custom.nginx.org/feature-b"}} # Insert config for feature B if the annotation is set # Print the value assigned to the annotation: {{$value}} {{end}} ``` Consider the following Ingress resource and note how we set two annotations: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress namespace: production annotations: custom.nginx.org/feature-a: "on" custom.nginx.org/feature-b: "512" spec: rules: - host: example.com . . . ``` Assuming that the Ingress Controller is using that customized template, it will generate a config for the Ingress resource that will include the following part, generated by our template excerpt: ```yaml # This is the configuration for cafe-ingress/default # Insert config for feature A if the annotation is set # Insert config for feature B if the annotation is set # Print the value assigned to the annotation: 512 ``` **Notes**: - You can customize the template to insert you custom annotations via [custom templates](/nic//configuration/global-configuration/custom-templates.md). - The Ingress Controller uses go templates to generate NGINX config. You can read more information about go templates [here](https://golang.org/pkg/text/template/). See the examples in the next section that use custom annotations to configure NGINX features. ### Custom Annotations with Mergeable Ingress Resources A Mergeable Ingress resource consists of multiple Ingress resources - one master and one or several minions. Read more about Mergeable Ingress resources [here](/nic/configuration/ingress-resources/cross-namespace-configuration.md). If you'd like to use custom annotations with Mergeable Ingress resources, please keep the following in mind: - Custom annotations can be used in the Master and in Minions. For Minions, you can access them in the template only when processing locations. If you access `$.Ingress` anywhere in the Ingress template, you will get the master Ingress resource. To access a Minion Ingress resource, use `$location.MinionIngress`. However, it is only available when processing locations: ```jinja2 {{range $location := $server.Locations}} location {{$location.Path}} { {{with $location.MinionIngress}} # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}} {{end}} } {{end}} ``` **Note**: `$location.MinionIngress` is a pointer. When a regular Ingress resource is processed in the template, the value of the pointer is `nil`. Thus, it is important that you check that `$location.MinionIngress` is not `nil` as in the example above using the `with` action. - Minions do not inherent custom annotations of the master. ### Helper Functions Helper functions can be used in the Ingress template to parse the values of custom annotations. | Function | Input Arguments | Return Arguments | Description | | ---| ---| ---| --- | | ``split`` | ``s, sep string`` | ``[]string`` | Splits the string ``s`` into a slice of strings separated by the ``sep``. | | ``trim`` | ``s string`` | ``string`` | Trims the trailing and leading whitespace from the string ``s``. | | ``contains`` | ``s, substr string`` | ``bool`` | Tests whether the string ``substr`` is a substring of the string ``s``. | | ``hasPrefix`` | ``s, prefix string`` | ``bool`` | Tests whether the string ``prefix`` is a prefix of the string ``s``. | | ``hasSuffix`` | ``s, suffix string`` | ``bool`` | Tests whether the string ``suffix`` is a suffix of the string ``s``. | | ``toLower`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their lower case. | | ``toUpper`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their upper case. | | ``replaceAll`` | ``s, old, new string`` | ``string`` | Replaces all occurrences of ``old`` with ``new`` in the string ``s``. | Consider the following custom annotation `custom.nginx.org/allowed-ips`, which expects a comma-separated list of IP addresses: ```yaml annotations: custom.nginx.org/allowed-ips: "192.168.1.3, 10.0.0.13" ``` The helper functions can parse the value of the `custom.nginx.org/allowed-ips` annotation, so that in the template you can use each IP address separately. Consider the following template excerpt: ```jinja2 {{range $ip := split (index $.Ingress.Annotations "custom.nginx.org/allowed-ips") ","}} allow {{trim $ip}}; {{end}} deny all; ``` The template excerpt will generate the following configuration: ``` allow 192.168.1.3; allow 10.0.0.13; deny all; ``` ## Example See the [custom annotations example](https://github.com/nginx/kubernetes-ingress/blob/v/examples/ingress-resources/custom-annotations). -------------------------------------------------------------------------------- title: "Custom annotations" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/ingress-resources/custom-annotations/index.md -------------------------------------------------------------------------------- This topic explains how you can use custom annotations with F5 NGINX Ingress Controller LTS. Custom annotations enable you to quickly extend the Ingress resource to support many advanced features of NGINX, such as rate limiting, caching, etc. ## Overview NGINX Ingress Controller LTS supports a number of annotations for the Ingress resource that fine tune NGINX configuration (for example, connection timeouts) or enable additional features (for example, JWT validation). The complete list of annotations is available [here](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations.md). The annotations are provided only for the most common features and use cases, meaning that not every NGINX feature or a customization option is available through the annotations. Additionally, even if an annotation is available, it might not give you the satisfactory level of control of a particular NGINX feature. Custom annotations allow you to add an annotation for an NGINX feature that is not available as a regular annotation. In contrast with regular annotations, to add a custom annotation, you don't need to modify the Ingress Controller source code -- just modify the template. Additionally, with a custom annotation, you get full control of how the feature is implemented in NGINX configuration. ## Usage The Ingress Controller generates NGINX configuration for Ingress resources by executing a configuration template. See [NGINX Plus template](https://github.com/nginx/kubernetes-ingress/blob/release-2026-lts/internal/configs/version1/nginx-plus.ingress.tmpl). To support custom annotations, the template has access to the information about the Ingress resource - its *name*, *namespace* and *annotations*. It is possible to check if a particular annotation present in the Ingress resource and conditionally insert NGINX configuration directives at multiple NGINX contexts - `http`, `server`, `location` or `upstream`. Additionally, you can get the value that is set to the annotation. Consider the following excerpt from the template, which was extended to support two custom annotations: ```jinja2 # This is the configuration for {{$.Ingress.Name}}/{{$.Ingress.Namespace}} {{if index $.Ingress.Annotations "custom.nginx.org/feature-a"}} # Insert config for feature A if the annotation is set {{end}} {{with $value := index $.Ingress.Annotations "custom.nginx.org/feature-b"}} # Insert config for feature B if the annotation is set # Print the value assigned to the annotation: {{$value}} {{end}} ``` Consider the following Ingress resource and note how we set two annotations: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress namespace: production annotations: custom.nginx.org/feature-a: "on" custom.nginx.org/feature-b: "512" spec: rules: - host: example.com . . . ``` Assuming that the Ingress Controller is using that customized template, it will generate a config for the Ingress resource that will include the following part, generated by our template excerpt: ```yaml # This is the configuration for cafe-ingress/default # Insert config for feature A if the annotation is set # Insert config for feature B if the annotation is set # Print the value assigned to the annotation: 512 ``` **Notes**: - You can customize the template to insert you custom annotations via [custom templates](/nic/lts/configuration/global-configuration/custom-templates.md). - The Ingress Controller uses go templates to generate NGINX config. You can read more information about go templates [here](https://golang.org/pkg/text/template/). See the examples in the next section that use custom annotations to configure NGINX features. ### Custom Annotations with Mergeable Ingress Resources A Mergeable Ingress resource consists of multiple Ingress resources - one master and one or several minions. Read more about Mergeable Ingress resources [here](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md). If you'd like to use custom annotations with Mergeable Ingress resources, please keep the following in mind: - Custom annotations can be used in the Master and in Minions. For Minions, you can access them in the template only when processing locations. If you access `$.Ingress` anywhere in the Ingress template, you will get the master Ingress resource. To access a Minion Ingress resource, use `$location.MinionIngress`. However, it is only available when processing locations: ```jinja2 {{range $location := $server.Locations}} location {{$location.Path}} { {{with $location.MinionIngress}} # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}} {{end}} } {{end}} ``` **Note**: `$location.MinionIngress` is a pointer. When a regular Ingress resource is processed in the template, the value of the pointer is `nil`. Thus, it is important that you check that `$location.MinionIngress` is not `nil` as in the example above using the `with` action. - Minions do not inherent custom annotations of the master. ### Helper Functions Helper functions can be used in the Ingress template to parse the values of custom annotations. | Function | Input Arguments | Return Arguments | Description | | ---| ---| ---| --- | | ``split`` | ``s, sep string`` | ``[]string`` | Splits the string ``s`` into a slice of strings separated by the ``sep``. | | ``trim`` | ``s string`` | ``string`` | Trims the trailing and leading whitespace from the string ``s``. | | ``contains`` | ``s, substr string`` | ``bool`` | Tests whether the string ``substr`` is a substring of the string ``s``. | | ``hasPrefix`` | ``s, prefix string`` | ``bool`` | Tests whether the string ``prefix`` is a prefix of the string ``s``. | | ``hasSuffix`` | ``s, suffix string`` | ``bool`` | Tests whether the string ``suffix`` is a suffix of the string ``s``. | | ``toLower`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their lower case. | | ``toUpper`` | ``s string`` | ``bool`` | Converts all letters in the string ``s`` to their upper case. | | ``replaceAll`` | ``s, old, new string`` | ``string`` | Replaces all occurrences of ``old`` with ``new`` in the string ``s``. | Consider the following custom annotation `custom.nginx.org/allowed-ips`, which expects a comma-separated list of IP addresses: ```yaml annotations: custom.nginx.org/allowed-ips: "192.168.1.3, 10.0.0.13" ``` The helper functions can parse the value of the `custom.nginx.org/allowed-ips` annotation, so that in the template you can use each IP address separately. Consider the following template excerpt: ```jinja2 {{range $ip := split (index $.Ingress.Annotations "custom.nginx.org/allowed-ips") ","}} allow {{trim $ip}}; {{end}} deny all; ``` The template excerpt will generate the following configuration: ``` allow 192.168.1.3; allow 10.0.0.13; deny all; ``` ## Example See the [custom annotations example](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/custom-annotations). -------------------------------------------------------------------------------- title: "DoS protected resource specification" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-dos/dos-protected/index.md -------------------------------------------------------------------------------- F5 DoS for NGINX protected resource specification **Note:** This feature is only available using the NGINX Plus [F5 DoS for NGINX Module](/nap-dos/deployment-guide/learn-about-deployment.md). ## DoS Protected resource specification Below is an example of a DoS protected resource. ```yaml apiVersion: appprotectdos.f5.com/v1beta1 kind: DosProtectedResource metadata: name: dos-protected spec: enable: true name: "my-dos" apDosMonitor: uri: "webapp.example.com" ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables F5 DoS for NGINX, Default value: false. | ``bool`` | No | |``name`` | Name of the protected object, max of 63 characters. | ``string`` | Yes | |``dosAccessLogDest`` | The log destination for the access log with dos log format. Accepted variables are ``:`` or ``stderr``. | ``string`` | No | |``apDosMonitor.uri`` | The destination to the desired protected object. [App Protect DoS monitor](#dosprotectedresourceapdosmonitor) Default value: None, URL will be extracted from the first request which arrives and taken from "Host" header or from destination ip+port. | ``string`` | No | |``apDosMonitor.protocol`` | Determines if the server listens on http1 / http2 / grpc / websocket. [App Protect DoS monitor](#dosprotectedresourceapdosmonitor) Default value: http1. | ``enum`` | No | |``apDosMonitor.timeout`` | Determines how long (in seconds) should F5 DoS for NGINX wait for a response. [App Protect DoS monitor](#dosprotectedresourceapdosmonitor) Default value: 10 seconds for http1/http2 and 5 seconds for grpc. | ``int64`` | No | |``apDosPolicy`` | The [App Protect DoS policy](#dosprotectedresourceapdospolicy) of the dos. Accepts an optional namespace. | ``string`` | No | |``dosSecurityLog.enable`` | Enables security log. | ``bool`` | No | |``dosSecurityLog.apDosLogConf`` | The [App Protect DoS log conf](/nic/integrations/app-protect-dos/configuration.md#app-protect-dos-logs) resource. Accepts an optional namespace. | ``string`` | No | |``dosSecurityLog.dosLogDest`` | The log destination for the security log. Accepted variables are ``:`` or ``stderr``. Default is ``127.0.0.1:514``. | ``string`` | No | |``allowList`` | List of allowed IP addresses and subnet masks. Each entry is represented by an `IPWithMask` string. | ``[]AllowListEntry`` | No | ### DosProtectedResource.apDosPolicy The `apDosPolicy` is a reference (qualified identifier in the format `namespace/name`) to the policy configuration defined as an `ApDosPolicy`. ### DosProtectedResource.apDosMonitor This is how F5 DoS for NGINX monitors the stress level of the protected object. The monitor requests are sent from localhost (127.0.0.1). ### Invalid DoS Protected resources NGINX will treat a DoS protected resource as invalid if one of the following conditions is met: - The DoS protected resource doesn't pass the [comprehensive validation](#comprehensive-validation). - The DoS protected resource isn't present in the cluster. ### Validation Two types of validation are available for the DoS protected resource: - *Structural validation*, done by `kubectl` and the Kubernetes API server. - *Comprehensive validation*, done by NGINX Ingress Controller. #### Structural validation The custom resource definition for the DoS protected resource includes a structural OpenAPI schema, which describes the type of every field of the resource. If you try to create (or update) a resource that violates the structural schema -- for example, the resource uses a string value instead of a bool in the `enable` field -- `kubectl` and the Kubernetes API server will reject the resource. - Example of `kubectl` validation: ```shell kubectl apply -f apdos-protected.yaml ``` ```shell error: error validating "examples/app-protect-dos/apdos-protected.yaml": error validating data: ValidationError(DosProtectedResource.spec.enable): invalid type for com.f5.appprotectdos.v1beta1.DosProtectedResource.spec.enable: got "string", expected "boolean"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f access-control-policy-allow.yaml --validate=false ``` ```shell The DosProtectedResource "dos-protected" is invalid: spec.enable: Invalid value: "string": spec.enable in body must be of type boolean: "string" ``` If a resource passes structural validation, then NGINX Ingress Controller will start comprehensive validation. #### Comprehensive validation NGINX Ingress Controller validates the fields of a DoS protected resource. If a resource is invalid, NGINX Ingress Controller will reject it. The resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. You can use `kubectl` to check if NGINX Ingress Controller successfully applied a DoS protected resource configuration. For our example `dos-protected` DoS protected resource, we can run: ```shell kubectl describe dosprotectedresource dos-protected ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 12s (x2 over 18h) nginx-ingress-controller Configuration for default/dos-protected was added or updated ``` Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a dos protected resource `dos-protected` with an invalid URI `bad` in the `dosSecurityLog/dosLogDest` field, you will get: ```shell kubectl describe policy webapp-policy ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 2s nginx-ingress-controller error validating DosProtectedResource: dos-protected invalid field: dosSecurityLog/dosLogDest err: invalid log destination: bad, must follow format: : or stderr ``` The events section has Warning event with the rejection error in the message. **Note:** If you invalidate an existing resource, NGINX Ingress Controller will reject it. -------------------------------------------------------------------------------- title: "Enable OpenTelemetry" url: https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/opentelemetry/index.md -------------------------------------------------------------------------------- This topic describes how to enable [OpenTelemetry](https://opentelemetry.io/) for F5 NGINX Ingress Controller using the [native NGINX module](https://nginx.org/en/docs/ngx_otel_module.html). ## Before you begin To complete this guide, you need the following pre-requisites: - An [NGINX Ingress Controller installation](/nic/install/) with OpenTelemetry (v5.1.0+) ## Load the OpenTelemetry module To enable OpenTelemetry, you must first load the module by adding the [_otel-exporter-endpoint_ ConfigMap key](/nic/configuration/global-configuration/configmap-resource.md#modules), which takes an endpoint argument. The following is an example of a OpenTelemetry collector running in your cluster as the target for exporting data: ```yaml otel-exporter-endpoint: "http://otel-collector.default.svc.cluster.local:4317" ``` A complete ConfigMap example with all OpenTelemetry options could look as follows: *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/shared-examples/otel/nginx-config.yaml]* ## Enable OpenTelemetry Once you have loaded the module, you can now enable OpenTelemetry. You can configure it globally for all resources, or on a per resource basis. ### Global To enable OpenTelemetry for all resources, set the _otel-trace-in-http_ ConfigMap key to `true`: ```yaml otel-trace-in-http: "true" ``` ### Per resource You can configure OpenTelemetry on a per resource basis in NGINX Ingress Controller. For this functionality, you must [enable snippets](/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md) with the `-enable-snippets` command-line argument. Based on the state of global configuration, you can selectively enable or disable metrics for each resource. #### Enable a specific resource or path With OpenTelemetry **disabled** globally, you can enable it for a specific resource using the server snippet annotation: ```yaml nginx.org/server-snippets: | otel_trace on; ``` You can enable it for specific paths using [Mergeable Ingress resources](/nic/configuration/ingress-resources/cross-namespace-configuration.md). Use the server snippet annotation for the paths of a specific Minion Ingress resource: ```yaml nginx.org/location-snippets: | otel_trace on; ``` #### Disable a specific resource or path With OpenTelemetry **enabled** globally, you can disable it for a specific resource using the server snippet annotation: ```yaml nginx.org/server-snippets: | otel_trace off; ``` You can disable it for specific paths using [Mergeable Ingress resources](/nic/configuration/ingress-resources/cross-namespace-configuration.md). Use the server snippet annotation for the paths of a specific Minion Ingress resource: ```yaml nginx.org/location-snippets: | otel_trace off; ``` ## Customize OpenTelemetry **Note:** You cannot modify the additional directives in the _otel_exporter_ block using snippets. You can customize OpenTelemetry through the supported [OpenTelemetry module directives](https://nginx.org/en/docs/ngx_otel_module.html). Use the `location-snippets` ConfigMap keys or annotations to insert those directives into the generated NGINX configuration. -------------------------------------------------------------------------------- title: "Enable OpenTelemetry" url: https://docs.nginx.com/nginx-ingress-controller/lts/logging-and-monitoring/opentelemetry/index.md -------------------------------------------------------------------------------- This topic describes how to enable [OpenTelemetry](https://opentelemetry.io/) for F5 NGINX Ingress Controller LTS using the [native NGINX module](https://nginx.org/en/docs/ngx_otel_module.html). ## Before you begin To complete this guide, you need the following pre-requisites: - An [NGINX Ingress Controller LTS installation](/nic/lts/install/) with OpenTelemetry (v5.1.0+) ## Load the OpenTelemetry module To enable OpenTelemetry, you must first load the module by adding the [_otel-exporter-endpoint_ ConfigMap key](/nic/lts/configuration/global-configuration/configmap-resource.md#modules), which takes an endpoint argument. The following is an example of a OpenTelemetry collector running in your cluster as the target for exporting data: ```yaml otel-exporter-endpoint: "http://otel-collector.default.svc.cluster.local:4317" ``` A complete ConfigMap example with all OpenTelemetry options could look as follows: *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/shared-examples/otel/nginx-config.yaml]* ## Enable OpenTelemetry Once you have loaded the module, you can now enable OpenTelemetry. You can configure it globally for all resources, or on a per resource basis. ### Global To enable OpenTelemetry for all resources, set the _otel-trace-in-http_ ConfigMap key to `true`: ```yaml otel-trace-in-http: "true" ``` ### Per resource You can configure OpenTelemetry on a per resource basis in NGINX Ingress Controller LTS. For this functionality, you must [enable snippets](/nic/lts/configuration/ingress-resources/advanced-configuration-with-snippets.md) with the `-enable-snippets` command-line argument. Based on the state of global configuration, you can selectively enable or disable metrics for each resource. #### Enable a specific resource or path With OpenTelemetry **disabled** globally, you can enable it for a specific resource using the server snippet annotation: ```yaml nginx.org/server-snippets: | otel_trace on; ``` You can enable it for specific paths using [Mergeable Ingress resources](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md). Use the server snippet annotation for the paths of a specific Minion Ingress resource: ```yaml nginx.org/location-snippets: | otel_trace on; ``` #### Disable a specific resource or path With OpenTelemetry **enabled** globally, you can disable it for a specific resource using the server snippet annotation: ```yaml nginx.org/server-snippets: | otel_trace off; ``` You can disable it for specific paths using [Mergeable Ingress resources](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md). Use the server snippet annotation for the paths of a specific Minion Ingress resource: ```yaml nginx.org/location-snippets: | otel_trace off; ``` ## Customize OpenTelemetry **note:** You cannot modify the additional directives in the _otel_exporter_ block using snippets. You can customize OpenTelemetry through the supported [OpenTelemetry module directives](https://nginx.org/en/docs/ngx_otel_module.html). Use the `location-snippets` ConfigMap keys or annotations to insert those directives into the generated NGINX configuration. -------------------------------------------------------------------------------- title: "Extensibility with NGINX Plus" url: https://docs.nginx.com/nginx-ingress-controller/lts/overview/nginx-plus/index.md -------------------------------------------------------------------------------- This document explains how F5 NGINX Plus can extend the functionality of F5 NGINX Ingress Controller LTS. NGINX Ingress Controller LTS works with [NGINX Plus](https://www.nginx.com/products/nginx/), a commercial closed source version of NGINX which has additional features and support from NGINX Inc. NGINX Ingress Controller LTS can leverage functionality from NGINX Plus to extend its base capabilities. --- ## Additional features - _Real-time metrics_: Metrics for NGINX Plus and application performance are available through the API or the [NGINX Status Page](/nic/lts/logging-and-monitoring/status-page). These metrics can also be exported to [Prometheus](/nic/lts/logging-and-monitoring/prometheus). - _Additional load balancing methods_: The `least_time` and `random two least_time` methods and their derivatives become available. The NGINX [`ngx_http_upstream_module` documentation](https://nginx.org/en/docs/http/ngx_http_upstream_module.html) has the complete list of load balancing methods. - _Session persistence_: While the *sticky cookie* method is available in both NGINX and NGINX Plus, NGINX Plus provides additional session persistence methods, including *sticky route* and *sticky learn*. See the [Ingress Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/session-persistence) and [Custom Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/session-persistence) examples. - _Active health checks_: See the [Ingress Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/health-checks) and [Custom Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/health-checks) examples. - _JWT validation_: See the [Ingress Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/jwt) and [Custom Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/jwt) examples. For a comprehensive guide of NGINX Plus features available with Ingress resources, see the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource) and [Annotations](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations) documentation. **note:** NGINX Plus features are configured for Ingress resources using Annotations that start with `nginx.com`. For a comprehensive guide of NGINX Plus features available with custom resources, see the [Policy](/nic/lts/configuration/policy-resource), [VirtualServer](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources) and [TransportServer](/nic/lts/configuration/transportserver-resource) documentation. --- ## Dynamic reconfiguration NGINX Ingress Controller LTS updates the configuration of the load balancer to reflect changes every time the number of pods exposed through an Ingress resource changes. When using NGINX, the configuration file must be changed then reloaded. For NGINX Plus, its dynamic reconfiguration is utilized, updating NGINX Plus without reloading. This avoids the increase of memory usage caused by reloads (Particularly with large volumes of client requests) and when load balancing applications with long-lived connections (Such as those using WebSockets or handling file uploads, downloads or streaming). -------------------------------------------------------------------------------- title: "Extensibility with NGINX Plus" url: https://docs.nginx.com/nginx-ingress-controller/overview/nginx-plus/index.md -------------------------------------------------------------------------------- This document explains how F5 NGINX Plus can extend the functionality of F5 NGINX Ingress Controller. NGINX Ingress Controller works with [NGINX](https://nginx.org/) as well as [NGINX Plus](https://www.nginx.com/products/nginx/), a commercial closed source version of NGINX which has additional features and support from NGINX Inc. NGINX Ingress Controller can leverage functionality from NGINX Plus to extend its base capabilities. --- ## Additional features - _Real-time metrics_: Metrics for NGINX Plus and application performance are available through the API or the [NGINX Status Page](/nic/logging-and-monitoring/status-page). These metrics can also be exported to [Prometheus](/nic/logging-and-monitoring/prometheus). - _Additional load balancing methods_: The `least_time` and `random two least_time` methods and their derivatives become available. The NGINX [`ngx_http_upstream_module` documentation](https://nginx.org/en/docs/http/ngx_http_upstream_module.html) has the complete list of load balancing methods. - _Session persistence_: While the *sticky cookie* method is available in both NGINX and NGINX Plus, NGINX Plus provides additional session persistence methods, including *sticky route* and *sticky learn*. See the [Ingress Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/session-persistence) and [Custom Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/session-persistence) examples. - _Active health checks_: See the [Ingress Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/health-checks) and [Custom Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/health-checks) examples. - _JWT validation_: See the [Ingress Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/jwt) and [Custom Resource](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/jwt) examples. For a comprehensive guide of NGINX Plus features available with Ingress resources, see the [ConfigMap](/nic/configuration/global-configuration/configmap-resource) and [Annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations) documentation. **Note:** NGINX Plus features are configured for Ingress resources using Annotations that start with `nginx.com`. For a comprehensive guide of NGINX Plus features available with custom resources, see the [Policy](/nic/configuration/policy-resource), [VirtualServer](/nic/configuration/virtualserver-and-virtualserverroute-resources) and [TransportServer](/nic/configuration/transportserver-resource) documentation. --- ## Dynamic reconfiguration NGINX Ingress Controller updates the configuration of the load balancer to reflect changes every time the number of pods exposed through an Ingress resource changes. When using NGINX, the configuration file must be changed then reloaded. For NGINX Plus, its dynamic reconfiguration is utilized, updating NGINX Plus without reloading. This avoids the increase of memory usage caused by reloads (Particularly with large volumes of client requests) and when load balancing applications with long-lived connections (Such as those using WebSockets or handling file uploads, downloads or streaming). -------------------------------------------------------------------------------- title: "Manage Policy resources with kubectl" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/policy-resource/using-policy/index.md -------------------------------------------------------------------------------- Use standard `kubectl` commands to work with Policy resources, just as you would with built-in Kubernetes resources. For example, the following command creates a Policy resource defined in `access-control-policy-allow.yaml` with the name `webapp-policy`: ```shell kubectl apply -f access-control-policy-allow.yaml policy.k8s.nginx.org/webapp-policy configured ``` Get the resource by running: ```shell kubectl get policy webapp-policy NAME AGE webapp-policy 27m ``` For `kubectl get` and similar commands, you can also use the short name `pol` instead of `policy`. ## Attach policies to a resource You can attach policies to VirtualServer, VirtualServerRoute, and Ingress resources. For example: - VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe spec: host: cafe.example.com tls: secret: cafe-secret policies: # spec policies - name: policy1 upstreams: - name: coffee service: coffee-svc port: 80 routes: - path: /tea policies: # route policies - name: policy2 namespace: cafe route: tea/tea - path: /coffee policies: # route policies - name: policy3 namespace: cafe action: pass: coffee ``` For VirtualServer, you can apply a policy: * to all routes (spec policies) * to a specific route (route policies) Route policies of the same type override spec policies. In the example above, if `policy-1` and `policy-3` are both `accessControl` policies, NGINX applies `policy-3` to requests for `cafe.example.com/coffee`. NGINX enforces this override: the spec policies apply in the `server` context of the configuration, and the route policies apply in the `location` context. As a result, the route policies of the same type take precedence. - VirtualServerRoute, referenced by the VirtualServer above: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: tea namespace: tea spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 subroutes: # subroute policies - path: /tea policies: - name: policy4 namespace: tea action: pass: tea ``` For VirtualServerRoute, you can apply a policy to a subroute (subroute policies). Subroute policies of the same type override spec policies. In the example above, if `policy-1` (in the VirtualServer) and `policy-4` are both `accessControl` policies, NGINX applies `policy-4` to requests for `cafe.example.com/tea`. As with the VirtualServer, NGINX enforces this override. Subroute policies always override route policies, regardless of type. For example, NGINX Ingress Controller LTS ignores `policy-2` from the VirtualServer route for the `/tea` subroute, because the subroute has its own policies, `policy4` in this case. If the subroute had no policies, NGINX Ingress Controller LTS would apply `policy-2` instead. NGINX Ingress Controller LTS enforces this override: the `location` context for the subroute has either route policies or subroute policies, but never both. - Ingress: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: nginx.org/policies: "webapp-policy" spec: ingressClassName: nginx tls: - hosts: - cafe.example.com secretName: tls-secret rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` For Ingress, you can apply policies: * to a single Ingress * to a master Ingress, where minion Ingresses inherit the policies * to minion Ingresses, where minion policies override master policies ## Invalid policies NGINX treats a policy as invalid if any of the following conditions is true: - The policy doesn't pass [comprehensive validation](#comprehensive-validation). - The policy isn't present in the cluster. - The policy doesn't meet its type-specific requirements. For example, an `ingressMTLS` policy requires TLS termination turned on in the VirtualServer. For an invalid policy, NGINX returns the 500 status code for client requests, following these rules: - If a policy is referenced in a VirtualServer `route` or a VirtualServerRoute `subroute`, NGINX returns the 500 status code for requests to the URIs of that route or subroute. - If a policy is referenced in the VirtualServer `spec`, NGINX returns the 500 status code for requests to all URIs of that VirtualServer. If a policy is invalid, the VirtualServer or VirtualServerRoute gets the [status](/nic/lts/configuration/global-configuration/reporting-resources-status.md#virtualserver-and-virtualserverroute-resources) state `Warning`, with a message that explains why the policy is invalid. ## Validation Two types of validation are available for the Policy resource: - *Structural validation*, done by `kubectl` and the Kubernetes API server. - *Comprehensive validation*, done by NGINX Ingress Controller LTS. ### Structural validation The custom resource definition for the Policy includes a structural OpenAPI schema, which describes the type of every field of the resource. If you try to create or update a resource that violates the structural schema, for example, if the resource uses a string value instead of an array of strings in the `allow` field, `kubectl` and the Kubernetes API server reject the resource. - Example of `kubectl` validation: ```shell kubectl apply -f access-control-policy-allow.yaml error: error validating "access-control-policy-allow.yaml": error validating data: ValidationError(Policy.spec.accessControl.allow): invalid type for org.nginx.k8s.v1.Policy.spec.accessControl.allow: got "string", expected "array"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f access-control-policy-allow.yaml --validate=false The Policy "webapp-policy" is invalid: spec.accessControl.allow: Invalid value: "string": spec.accessControl.allow in body must be of type array: "string" ``` If a resource passes structural validation, NGINX Ingress Controller LTS's comprehensive validation runs next. ### Comprehensive validation NGINX Ingress Controller LTS validates the fields of a Policy resource. If a resource is invalid, NGINX Ingress Controller LTS rejects it. The resource continues to exist in the cluster, but NGINX Ingress Controller LTS ignores it. Use `kubectl` to check whether NGINX Ingress Controller LTS successfully applied a Policy configuration. For the example `webapp-policy` Policy, run: ```shell kubectl describe pol webapp-policy . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated ``` The `events` section includes a `Normal` event with the `AddedOrUpdated` reason, which tells you the configuration applied successfully. If you create an invalid resource, NGINX Ingress Controller LTS rejects it and emits a Rejected event. For example, if you create a Policy `webapp-policy` with an invalid IP `10.0.0.` in the `allow` field, you get: ```shell kubectl describe policy webapp-policy . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 7s nginx-ingress-controller Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP ``` The events section includes a `Warning` event with the `Rejected` reason. This information is also available in the `status` field of the Policy resource. Note the Status section of the Policy: ```shell kubectl describe pol webapp-policy . . . Status: Message: Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP Reason: Rejected State: Invalid ``` **Note:** If you make an existing resource invalid, NGINX Ingress Controller LTS rejects it. -------------------------------------------------------------------------------- title: "Management ConfigMap resource" url: https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/mgmt-configmap-resource/index.md -------------------------------------------------------------------------------- When using F5 NGINX Ingress Controller with NGINX Plus, it is required to pass a [command line argument](/nic/configuration/global-configuration/command-line-arguments) to NGINX Ingress Controller, `--mgmt-configmap=` which specifies the ConfigMap to use. The minimal required ConfigMap must have a `license-token-secret-name` key. Helm users will not need to create this map or pass the argument, it will be created with a Helm install. --- 1. Create a ConfigMap file with the name *nginx-config-mgmt.yaml* and set the values that make sense for your setup: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-config-mgmt namespace: nginx-ingress data: license-token-secret-name: "license-token" ``` 1. Create a new (or update the existing) ConfigMap resource: ```shell kubectl apply -f nginx-config-mgmt.yaml ``` The [NGINX Management](https://nginx.org/en/docs/ngx_mgmt_module.html) block configuration will be updated. ## Management ConfigMap keys |ConfigMap Key | Description | Default | | ---| ---| ---| |*license-token-secret-name* | Configures the secret used in the [license_token](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.com/license` with the base64 encoded JWT in the `license.jwt` key. | N/A | |*ssl-verify* | Configures the [ssl_verify](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_verify) directive, which enables or disables verification of the usage reporting endpoint certificate. | `true` | |*enforce-initial-report* | Configures the [enforce_initial_report](https://nginx.org/en/docs/ngx_mgmt_module.html#enforce_initial_report) directive, which enables or disables the 180-day grace period for sending the initial usage report. | `false` | |*usage-report-endpoint* | Configures the endpoint of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This is used to configure the endpoint NGINX uses to send usage reports to NIM. | `product.connect.nginx.com` | |*usage-report-interval* | Configures the interval of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This specifies the frequency that usage reports are sent. This field takes an [NGINX time](https://nginx.org/en/docs/syntax.html). | `1h` | |*usage-report-proxy-host* | Configures the host name of the [proxy](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy) directive with optional port. | N/A | |*ssl-trusted-certificate-secret-name* | Configures the secret used to create the file(s) referenced the in [ssl_trusted_certifcate](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_trusted_certificate), and [ssl_crl](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.org/ca`, where the `ca.crt` key contains a base64 encoded trusted cert, and the optional `ca.crl` key can contain a base64 encoded CRL. If the optional `ca.crl` key is supplied, it will configure the NGINX `ssl_crl` directive. | N/A | |*ssl-certificate-secret-name* | Configures the secret used to create the `ssl_certificate` and `ssl_certificate_key` directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `kubernetes.io/tls`| N/A | |*resolver-addresses* | Configures addresses used in the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive. This field takes a comma separated list of addresses. | N/A | |*resolver-ipv6* | Configures whether the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will look up IPv6 addresses. | `true` | |*resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will override the TTL value of responses from nameservers with. | N/A | -------------------------------------------------------------------------------- title: "Management ConfigMap resource" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/global-configuration/mgmt-configmap-resource/index.md -------------------------------------------------------------------------------- When using F5 NGINX Ingress Controller LTS, it is required to pass a [command line argument](/nic/lts/configuration/global-configuration/command-line-arguments) to NGINX Ingress Controller LTS, `--mgmt-configmap=` which specifies the ConfigMap to use. The minimal required ConfigMap must have a `license-token-secret-name` key. Helm users will not need to create this map or pass the argument, it will be created with a Helm install. --- 1. Create a ConfigMap file with the name *nginx-config-mgmt.yaml* and set the values that make sense for your setup: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-config-mgmt namespace: nginx-ingress data: license-token-secret-name: "license-token" ``` 1. Create a new (or update the existing) ConfigMap resource: ```shell kubectl apply -f nginx-config-mgmt.yaml ``` The [NGINX Management](https://nginx.org/en/docs/ngx_mgmt_module.html) block configuration will be updated. ## Management ConfigMap keys |ConfigMap Key | Description | Default | | ---| ---| ---| |*license-token-secret-name* | Configures the secret used in the [license_token](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive. This key assumes the secret is in the Namespace that NGINX Ingress Controller LTS is deployed in. The secret must be of type `nginx.com/license` with the base64 encoded JWT in the `license.jwt` key. | N/A | |*ssl-verify* | Configures the [ssl_verify](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_verify) directive, which enables or disables verification of the usage reporting endpoint certificate. | `true` | |*enforce-initial-report* | Configures the [enforce_initial_report](https://nginx.org/en/docs/ngx_mgmt_module.html#enforce_initial_report) directive, which enables or disables the 180-day grace period for sending the initial usage report. | `false` | |*usage-report-endpoint* | Configures the endpoint of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This is used to configure the endpoint NGINX uses to send usage reports to NIM. | `product.connect.nginx.com` | |*usage-report-interval* | Configures the interval of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This specifies the frequency that usage reports are sent. This field takes an [NGINX time](https://nginx.org/en/docs/syntax.html). | `1h` | |*usage-report-proxy-host* | Configures the host name of the [proxy](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy) directive with optional port. | N/A | |*ssl-trusted-certificate-secret-name* | Configures the secret used to create the file(s) referenced the in [ssl_trusted_certifcate](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_trusted_certificate), and [ssl_crl](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller LTS is deployed in. The secret must be of type `nginx.org/ca`, where the `ca.crt` key contains a base64 encoded trusted cert, and the optional `ca.crl` key can contain a base64 encoded CRL. If the optional `ca.crl` key is supplied, it will configure the NGINX `ssl_crl` directive. | N/A | |*ssl-certificate-secret-name* | Configures the secret used to create the `ssl_certificate` and `ssl_certificate_key` directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller LTS is deployed in. The secret must be of type `kubernetes.io/tls`| N/A | |*resolver-addresses* | Configures addresses used in the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive. This field takes a comma separated list of addresses. | N/A | |*resolver-ipv6* | Configures whether the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will look up IPv6 addresses. | `true` | |*resolver-valid* | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will override the TTL value of responses from nameservers with. | N/A | -------------------------------------------------------------------------------- title: "Security recommendations" url: https://docs.nginx.com/nginx-ingress-controller/configuration/security/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller follows Kubernetes best practices: this page outlines configuration specific to NGINX Ingress Controller you may require, including links to examples in the [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v/examples). For general guidance, we recommend the official Kubernetes documentation for [Securing a Cluster](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/). ## Kubernetes recommendations ### RBAC and Service Accounts Kubernetes uses [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to control the resources and operations available to different types of users. NGINX Ingress Controller requires RBAC to configure a [ServiceUser](https://kubernetes.io/docs/concepts/security/service-accounts/#default-service-accounts), and provides least privilege access in its standard deployment configurations: - [Helm](https://github.com/nginx/kubernetes-ingress/blob/v/deployments/rbac/rbac.yaml) - [Manifests](https://github.com/nginx/kubernetes-ingress/blob/v/deployments/rbac/rbac.yaml) By default, the ServiceAccount has access to all Secret resources in the cluster. ### Secrets [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) are required by NGINX Ingress Controller for certificates and privacy keys, which Kubernetes stores unencrypted by default. We recommend following the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) to store these Secrets using at-rest encryption. ## NGINX Ingress Controller recommendations ### Configure root filesystem as read-only **Note:** This feature has its own documentation in [F5 WAF for NGINX](/waf/configure/kubernetes-read-only.md) documentation. It is compatible with a Kubernetes deployment: it **is not** compatible with [F5 DoS for NGINX](/nap-dos/). NGINX Ingress Controller is designed to be resilient against attacks in various ways, such as running the service as non-root to avoid changes to files. We recommend setting filesystems on all containers to read-only, this includes `nginx-ingress-controller`, though also includes `waf-enforcer` and `waf-config-mgr` when F5 WAF for NGINXv5 is in use. This is so that the attack surface is further reduced by limiting changes to binaries and libraries. This is not enabled by default, but can be enabled with **Helm** using the [**readOnlyRootFilesystem**](/nic/install/helm.md#configuration) argument in security contexts on all containers: `nginx-ingress-controller`, `waf_enforcer` and `waf_config_mgr`. For **Manifests**, uncomment the following sections of the deployment and add sections for `waf-enforcer` and `waf-config-mgr` containers: - `readOnlyRootFilesystem: true` - The entire **volumeMounts** section - The entire **initContainers** section The block below shows the code you will look for: ```yaml # volumes: # - name: nginx-etc # emptyDir: {} # - name: nginx-cache # do not set this value in statefulset if volumeclaimtemplate is set # emptyDir: {} # do not set this value in statefulset if volumeclaimtemplate is set # - name: nginx-lib # emptyDir: {} # - name: nginx-lib-state # emptyDir: {} # - name: nginx-log # emptyDir: {} . . . # readOnlyRootFilesystem: true . . . # volumeMounts: # - mountPath: /etc/nginx # name: nginx-etc # - mountPath: /var/cache/nginx # name: nginx-cache # - mountPath: /var/lib/nginx # name: nginx-lib # - mountPath: /var/lib/nginx/state # name: nginx-lib-state # - mountPath: /var/log/nginx # name: nginx-log ``` - Add **waf-enforcer** and **waf-config-mgr** container sections - Add `readOnlyFilesystem: true` in both containers security context sections ### Prometheus If Prometheus metrics are [enabled](/nic/logging-and-monitoring/prometheus.md), we recommend [using HTTPS](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-prometheus-tls-secret). ### Snippets Snippets allow raw NGINX configuration to be inserted into resources. They are intended for advanced NGINX users and could create vulnerabilities in a cluster if misused. Snippets are disabled by default. To use snippets, set the [**enable-snippets**]() command-line argument. **Note:** Snippets are **always** enabled for ConfigMap. For more information, read the following: - [Advanced configuration using Snippets](/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md) - [Using Snippets with VirtualServer/VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#using-snippets) - [Using Snippets with TransportServer](/nic/configuration/transportserver-resource.md#using-snippets) - [ConfigMap snippets and custom templates](/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) -------------------------------------------------------------------------------- title: "Security recommendations" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/security/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller LTS follows Kubernetes best practices: this page outlines configuration specific to NGINX Ingress Controller LTS you may require, including links to examples in the [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v/examples). For general guidance, we recommend the official Kubernetes documentation for [Securing a Cluster](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/). ## Kubernetes recommendations ### RBAC and Service Accounts Kubernetes uses [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to control the resources and operations available to different types of users. NGINX Ingress Controller LTS requires RBAC to configure a [ServiceUser](https://kubernetes.io/docs/concepts/security/service-accounts/#default-service-accounts), and provides least privilege access in its standard deployment configurations: - [Helm](https://github.com/nginx/kubernetes-ingress/blob/v/deployments/rbac/rbac.yaml) - [Manifests](https://github.com/nginx/kubernetes-ingress/blob/v/deployments/rbac/rbac.yaml) By default, the ServiceAccount has access to all Secret resources in the cluster. ### Secrets [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) are required by NGINX Ingress Controller LTS for certificates and privacy keys, which Kubernetes stores unencrypted by default. We recommend following the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) to store these Secrets using at-rest encryption. ## NGINX Ingress Controller LTS recommendations ### Configure root filesystem as read-only NGINX Ingress Controller LTS is designed to be resilient against attacks in various ways, such as running the service as non-root to avoid changes to files. We recommend setting filesystems on `nginx-ingress-controller` container to read-only. This is so that the attack surface is further reduced by limiting changes to binaries and libraries. This is not enabled by default, but can be enabled with **Helm** using the [**readOnlyRootFilesystem**](/nic/lts/install/helm.md#configuration) argument in security contexts. For **Manifests**, uncomment the following sections of the deployment: - `readOnlyRootFilesystem: true` - The entire **volumeMounts** section - The entire **initContainers** section The block below shows the code you will look for: ```yaml # volumes: # - name: nginx-etc # emptyDir: {} # - name: nginx-cache # do not set this value in statefulset if volumeclaimtemplate is set # emptyDir: {} # do not set this value in statefulset if volumeclaimtemplate is set # - name: nginx-lib # emptyDir: {} # - name: nginx-lib-state # emptyDir: {} # - name: nginx-log # emptyDir: {} . . . # readOnlyRootFilesystem: true . . . # volumeMounts: # - mountPath: /etc/nginx # name: nginx-etc # - mountPath: /var/cache/nginx # name: nginx-cache # - mountPath: /var/lib/nginx # name: nginx-lib # - mountPath: /var/lib/nginx/state # name: nginx-lib-state # - mountPath: /var/log/nginx # name: nginx-log ``` ### Prometheus If Prometheus metrics are [enabled](/nic/lts/logging-and-monitoring/prometheus.md), we recommend [using HTTPS](/nic/lts/configuration/global-configuration/command-line-arguments.md#cmdoption-prometheus-tls-secret). ### Snippets Snippets allow raw NGINX configuration to be inserted into resources. They are intended for advanced NGINX users and could create vulnerabilities in a cluster if misused. Snippets are disabled by default. To use snippets, set the [**enable-snippets**]() command-line argument. **caution:** Snippets are **always** enabled for ConfigMap. For more information, read the following: - [Advanced configuration using Snippets](/nic/lts/configuration/ingress-resources/advanced-configuration-with-snippets.md) - [Using Snippets with VirtualServer/VirtualServerRoute](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#using-snippets) - [Using Snippets with TransportServer](/nic/lts/configuration/transportserver-resource.md#using-snippets) - [ConfigMap snippets and custom templates](/nic/lts/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) -------------------------------------------------------------------------------- title: "Troubleshooting Ingress resources" url: https://docs.nginx.com/nginx-ingress-controller/lts/troubleshooting/troubleshoot-ingress/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot NGINX Ingress Controller LTS Policy Resources. ## Ingress resources After you create or update an Ingress resource, you can immediately check if the NGINX configuration for that Ingress resource was successfully applied by NGINX: ```shell kubectl describe ing cafe-ingress ``` ```shell Name: cafe-ingress Namespace: default Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 12s nginx-ingress-controller Configuration for default/cafe-ingress was added or updated ``` The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. -------------------------------------------------------------------------------- title: "Troubleshooting Ingress resources" url: https://docs.nginx.com/nginx-ingress-controller/troubleshooting/troubleshoot-ingress/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot NGINX Ingress Controller Policy Resources. ## Ingress resources After you create or update an Ingress resource, you can immediately check if the NGINX configuration for that Ingress resource was successfully applied by NGINX: ```shell kubectl describe ing cafe-ingress ``` ```shell Name: cafe-ingress Namespace: default Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 12s nginx-ingress-controller Configuration for default/cafe-ingress was added or updated ``` The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. -------------------------------------------------------------------------------- title: "Troubleshooting Policy resources" url: https://docs.nginx.com/nginx-ingress-controller/lts/troubleshooting/troubleshoot-configmap-policy/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot NGINX Ingress Controller LTS Policy Resources. ## Policy Resources After you create or update a Policy resource, you can use `kubectl describe` to check whether or not NGINX Ingress Controller LTS accepted the policy: ```shell kubectl describe pol webapp-policy ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated ``` The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. However, the fact that a policy was accepted doesn’t guarantee that the NGINX configuration was successfully applied. To verify the configuration applied, check the events of the [VirtualServer and VirtualServerRoute resources](/nic/lts/troubleshooting/troubleshoot-virtualserver.md) that reference the policy. ## ConfigMap Resources After you update the ConfigMap resource, you can immediately check if the configuration was successfully applied by NGINX: ```shell kubectl describe configmap nginx-config -n nginx-ingress Name: nginx-config Namespace: nginx-ingress Labels: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Updated 11s (x2 over 26m) nginx-ingress-controller Configuration from nginx-ingress/nginx-config was updated ``` Similar to *Policies*, the events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. -------------------------------------------------------------------------------- title: "Troubleshooting Policy resources" url: https://docs.nginx.com/nginx-ingress-controller/troubleshooting/troubleshoot-configmap-policy/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot NGINX Ingress Controller Policy Resources. ## Policy Resources After you create or update a Policy resource, you can use `kubectl describe` to check whether or not NGINX Ingress Controller accepted the policy: ```shell kubectl describe pol webapp-policy ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated ``` The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. However, the fact that a policy was accepted doesn’t guarantee that the NGINX configuration was successfully applied. To verify the configuration applied, check the events of the [VirtualServer and VirtualServerRoute resources](/nic/troubleshooting/troubleshoot-virtualserver.md) that reference the policy. ## ConfigMap Resources After you update the ConfigMap resource, you can immediately check if the configuration was successfully applied by NGINX: ```shell kubectl describe configmap nginx-config -n nginx-ingress Name: nginx-config Namespace: nginx-ingress Labels: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Updated 11s (x2 over 26m) nginx-ingress-controller Configuration from nginx-ingress/nginx-config was updated ``` Similar to *Policies*, the events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. -------------------------------------------------------------------------------- title: "Advanced configuration with Snippets" url: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-snippets/index.md -------------------------------------------------------------------------------- Snippets allow you to insert raw NGINX config into different contexts of the NGINX configurations that F5 NGINX Ingress Controller generates. Snippets are intended for advanced NGINX users who need more control over the generated NGINX configuration, and can be used in cases where Annotations and ConfigMap entries would not apply. ## Disadvantages of snippets Snippets are configured [using Annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#snippets-and-custom-templates), but are disabled by default due to their complexity. They are also available through the [ConfigMap](/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) resource. To use snippets, set the [`enable-snippets`](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets) command-line argument. Snippets have the following disadvantages: - *Complexity*. Snippets require you to: - Understand NGINX configuration primitives and implement a correct NGINX configuration. - Understand how NGINX Ingress Controller generates NGINX configuration so that a snippet doesn't interfere with the other features in the configuration. - *Decreased robustness*. An incorrect snippet can invalidate NGINX configuration, causing reload failures. Until the snippet is fixed, it will prevent any new configuration updates, including updates for the other Ingress resources. - *Security implications*. Snippets give access to NGINX configuration primitives, which are not validated by NGINX Ingress Controller. For example, a snippet can configure NGINX to serve the TLS certificates and keys used for TLS termination for Ingress resources. **Note:** If the NGINX configuration includes an invalid snippet, NGINX will continue to operate with the last valid configuration. ## Using snippets The example below shows how to use snippets to customize the NGINX configuration template using annotations. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress-with-snippets annotations: nginx.org/server-snippets: | location / { return 302 /coffee; } nginx.org/location-snippets: | add_header my-test-header test-value; spec: rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` These snippets generate the following NGINX configuration: **Note:** The example is shortened for conciseness. ```nginx server { listen 80; location / { return 302 /coffee; } location /coffee { proxy_http_version 1.1; add_header my-test-header test-value; ... proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-coffee-svc-80; } location /tea { proxy_http_version 1.1; add_header my-test-header test-value; ... proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-tea-svc-80; } } ``` ## Troubleshooting If a snippet includes an invalid NGINX configuration, NGINX Ingress Controller will fail to reload NGINX. The error will be reported in NGINX Ingress Controller logs and an event with the error will be associated with the Ingress resource: An example of an error from the logs: ```text [emerg] 31#31: unknown directive "badd_header" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54 Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress-with-snippets", UID:"f9656dc9-63a6-41dd-a499-525b0e0309bb", APIVersion:"extensions/v1beta1", ResourceVersion:"2322030", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" finished with error: exit status 1 ``` An example of an event with an error (you can view events associated with the Ingress by running `kubectl describe -n nginx-ingress ingress nginx-ingress`): ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 52m (x3 over 61m) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated finished with error: exit status 1 Warning AddedOrUpdatedWithError 54s (x2 over 89s) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" finished with error: exit status 1 ``` Additionally, to help troubleshoot snippets, a number of Prometheus metrics show the stats about failed reloads – `controller_nginx_last_reload_status` and `controller_nginx_reload_errors_total`. -------------------------------------------------------------------------------- title: "Advanced configuration with Snippets" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/ingress-resources/advanced-configuration-with-snippets/index.md -------------------------------------------------------------------------------- Snippets allow you to insert raw NGINX config into different contexts of the NGINX configurations that F5 NGINX Ingress Controller LTS generates. Snippets are intended for advanced NGINX users who need more control over the generated NGINX configuration, and can be used in cases where Annotations and ConfigMap entries would not apply. ## Disadvantages of snippets Snippets are configured [using Annotations](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations.md#snippets-and-custom-templates), but are disabled by default due to their complexity. They are also available through the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) resource. To use snippets, set the [`enable-snippets`](/nic/lts/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets) command-line argument. Snippets have the following disadvantages: - *Complexity*. Snippets require you to: - Understand NGINX configuration primitives and implement a correct NGINX configuration. - Understand how NGINX Ingress Controller LTS generates NGINX configuration so that a snippet doesn't interfere with the other features in the configuration. - *Decreased robustness*. An incorrect snippet can invalidate NGINX configuration, causing reload failures. Until the snippet is fixed, it will prevent any new configuration updates, including updates for the other Ingress resources. - *Security implications*. Snippets give access to NGINX configuration primitives, which are not validated by NGINX Ingress Controller LTS. For example, a snippet can configure NGINX to serve the TLS certificates and keys used for TLS termination for Ingress resources. **note:** If the NGINX configuration includes an invalid snippet, NGINX will continue to operate with the last valid configuration. ## Using snippets The example below shows how to use snippets to customize the NGINX configuration template using annotations. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress-with-snippets annotations: nginx.org/server-snippets: | location / { return 302 /coffee; } nginx.org/location-snippets: | add_header my-test-header test-value; spec: rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` These snippets generate the following NGINX configuration: **note:** The example is shortened for conciseness. ```nginx server { listen 80; location / { return 302 /coffee; } location /coffee { proxy_http_version 1.1; add_header my-test-header test-value; ... proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-coffee-svc-80; } location /tea { proxy_http_version 1.1; add_header my-test-header test-value; ... proxy_pass http://default-cafe-ingress-with-snippets-cafe.example.com-tea-svc-80; } } ``` ## Troubleshooting If a snippet includes an invalid NGINX configuration, NGINX Ingress Controller LTS will fail to reload NGINX. The error will be reported in NGINX Ingress Controller LTS logs and an event with the error will be associated with the Ingress resource: An example of an error from the logs: ```text [emerg] 31#31: unknown directive "badd_header" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54 Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress-with-snippets", UID:"f9656dc9-63a6-41dd-a499-525b0e0309bb", APIVersion:"extensions/v1beta1", ResourceVersion:"2322030", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" finished with error: exit status 1 ``` An example of an event with an error (you can view events associated with the Ingress by running `kubectl describe -n nginx-ingress ingress nginx-ingress`): ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 52m (x3 over 61m) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated finished with error: exit status 1 Warning AddedOrUpdatedWithError 54s (x2 over 89s) nginx-ingress-controller Configuration for default/cafe-ingress-with-snippets was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress-with-snippets: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" stderr: "nginx: [emerg] unknown directive \"badd_header\" in /etc/nginx/conf.d/default-cafe-ingress-with-snippets.conf:54\n" finished with error: exit status 1 ``` Additionally, to help troubleshoot snippets, a number of Prometheus metrics show the stats about failed reloads – `controller_nginx_last_reload_status` and `controller_nginx_reload_errors_total`. -------------------------------------------------------------------------------- title: "Configuration examples" url: https://docs.nginx.com/nginx-ingress-controller/configuration/configuration-examples/index.md -------------------------------------------------------------------------------- Our [GitHub repo](https://github.com/nginx/kubernetes-ingress) includes a number of configuration examples: - [*Examples of Custom Resources*](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources) show how to advanced NGINX features by using VirtualServer, VirtualServerRoute, TransportServer and Policy Custom Resources. - [*Examples of Ingress Resources*](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources) show how to use advanced NGINX features in Ingress resources with annotations. -------------------------------------------------------------------------------- title: "Configuration examples" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/configuration-examples/index.md -------------------------------------------------------------------------------- Our [GitHub repo](https://github.com/nginx/kubernetes-ingress) includes a number of configuration examples: - [*Examples of Custom Resources*](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/custom-resources) show how to advanced NGINX features by using VirtualServer, VirtualServerRoute, TransportServer and Policy Custom Resources. - [*Examples of Ingress Resources*](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources) show how to use advanced NGINX features in Ingress resources with annotations. -------------------------------------------------------------------------------- title: "Enable Prometheus metrics" url: https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/prometheus/index.md -------------------------------------------------------------------------------- This topic describes how to enable [Prometheus metrics](https://prometheus.io/) for F5 NGINX Ingress Controller. The metrics exposed include NGINX Ingress Controller data, as well as NGINX Open Source and NGINX Plus. ## Enabling Metrics ### Using Helm To enable Prometheus metrics when using *Helm* to install NGINX Ingress Controller, configure the `prometheus.*` parameters of the Helm chart. See the [Installation with Helm](/nic/install/helm.md) topic. #### Using ServiceMonitor When deploying with *Helm*, you can deploy a `Service` and `ServiceMonitor` resource using the `prometheus.service.*` and `prometheus.serviceMonitor.*` parameters. When these resources are deployed, Prometheus metrics exposed by NGINX Ingress Controller can be captured and enumerated using a `Prometheus` resource alongside a Prometheus Operator deployment. To view metrics captured this way, you will need: * A working [Prometheus resource and Prometheus Operator](https://prometheus-operator.dev/docs/getting-started/introduction/) * The latest ServiceMonitor CRD from the [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator) repository: ```shell LATEST=$(curl -s https://api.github.com/repos/prometheus-operator/prometheus-operator/releases/latest | jq -cr .tag_name) curl https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/$LATEST/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml | kubectl create -f - ``` ### Using Manifests If you're using *Kubernetes manifests* (Deployment, DaemonSet, or StatefulSet) to install the Ingress Controller, to enable Prometheus metrics: 1. Run the Ingress Controller with the `-enable-prometheus-metrics` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md). As a result, the Ingress Controller will expose NGINX or NGINX Plus metrics in the Prometheus format via the path `/metrics` on port `9113` (customizable via the `-prometheus-metrics-listen-port` command-line argument). 1. To enable TLS for the Prometheus endpoint, configure the `-prometheus-tls-secret` cli argument with the namespace and name of a TLS Secret. 1. Add the Prometheus port to the list of the ports of the Ingress Controller container in the template of the Ingress Controller pod: ```yaml - name: prometheus containerPort: 9113 ``` 1. Make Prometheus aware of the Ingress Controller targets by adding the following annotations to the template of the Ingress Controller pod (note: this assumes your Prometheus is configured to discover targets by analyzing the annotations of pods): ```yaml annotations: prometheus.io/scrape: "true" prometheus.io/port: "9113" prometheus.io/scheme: http ``` ## Available Metrics The Ingress Controller exports the following metrics: - NGINX/NGINX Plus metrics: - Exported by NGINX/NGINX Plus. Refer to the [NGINX Prometheus Exporter developer docs](https://github.com/nginx/nginx-prometheus-exporter#exported-metrics) to find more information about the exported metrics. - There is a Grafana dashboard for NGINX Plus metrics located in the root repo folder. - Calculated by the Ingress Controller: - `controller_upstream_server_response_latency_ms_count`. Bucketed response times from when NGINX establishes a connection to an upstream server to when the last byte of the response body is received by NGINX. **Note**: The metric for the upstream isn't available until traffic is sent to the upstream. The metric isn't enabled by default. To enable the metric, set the `-enable-latency-metrics` command-line argument. - Ingress Controller metrics - `controller_nginx_reloads_total`. Number of successful NGINX reloads. This includes the label `reason` with 2 possible values `endpoints` (the reason for the reload was an endpoints update) and `other` (the reload was caused by something other than an endpoint update like an ingress update). - `controller_nginx_reload_errors_total`. Number of unsuccessful NGINX reloads. - `controller_nginx_last_reload_status`. Status of the last NGINX reload, 0 meaning down and 1 up. - `controller_nginx_last_reload_milliseconds`. Duration in milliseconds of the last NGINX reload. - `controller_nginx_worker_processes_total`. Number of NGINX worker processes. This metric includes the constant label `generation` with two possible values `old` (the shutting down processes of the old generations) or `current` (the processes of the current generation). - `controller_ingress_resources_total`. Number of handled Ingress resources. This metric includes the label type, that groups the Ingress resources by their type (regular, [minion or master](/nic/configuration/ingress-resources/cross-namespace-configuration.md). **Note**: The metric doesn't count minions without a master. - `controller_virtualserver_resources_total`. Number of handled VirtualServer resources. - `controller_virtualserverroute_resources_total`. Number of handled VirtualServerRoute resources. **Note**: The metric counts only VirtualServerRoutes that have a reference from a VirtualServer. - `location_zone` (upstream services) metrics: - `location_zone_sent`. Number of bytes sent to clients. - `location_zone_received`. Number of bytes received from clients. - `location_zone_requests`. Total number of client requests. - `location_zone_responses`. Total number of responses sent to clients. - `location_zone_responses_codes`. Total number of responses sent to clients. - `location_zone_sent`. Number of bytes sent to clients. - `controller_transportserver_resources_total`. Number of handled TransportServer resources. This metric includes the label type, that groups the TransportServer resources by their type (passthrough, tcp or udp). - Workqueue metrics. **Note**: the workqueue is a queue used by the Ingress Controller to process changes to the relevant resources in the cluster like Ingress resources. The Ingress Controller uses only one queue. The metrics for that queue will have the label `name="taskQueue"` - `workqueue_depth`. Current depth of the workqueue. - `workqueue_queue_duration_second`. How long in seconds an item stays in the workqueue before being requested. - `workqueue_work_duration_seconds`. How long in seconds processing an item from the workqueue takes. **Note**: all metrics have the namespace `nginx_ingress`. For example, `nginx_ingress_controller_nginx_reloads_total`. **Note**: all metrics include the label `class`, which is set to the class of the Ingress Controller. The class is configured via the `-ingress-class` command-line argument. -------------------------------------------------------------------------------- title: "Enable Prometheus metrics" url: https://docs.nginx.com/nginx-ingress-controller/lts/logging-and-monitoring/prometheus/index.md -------------------------------------------------------------------------------- This topic describes how to enable [Prometheus metrics](https://prometheus.io/) for F5 NGINX Ingress Controller LTS. The metrics exposed include NGINX Ingress Controller LTS data and NGINX Plus metrics. ## Enabling Metrics ### Using Helm To enable Prometheus metrics when using *Helm* to install NGINX Ingress Controller LTS, configure the `prometheus.*` parameters of the Helm chart. See the [Installation with Helm](/nic/lts/install/helm.md) topic. #### Using ServiceMonitor When deploying with *Helm*, you can deploy a `Service` and `ServiceMonitor` resource using the `prometheus.service.*` and `prometheus.serviceMonitor.*` parameters. When these resources are deployed, Prometheus metrics exposed by NGINX Ingress Controller LTS can be captured and enumerated using a `Prometheus` resource alongside a Prometheus Operator deployment. To view metrics captured this way, you will need: * A working [Prometheus resource and Prometheus Operator](https://prometheus-operator.dev/docs/getting-started/introduction/) * The latest ServiceMonitor CRD from the [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator) repository: ```shell LATEST=$(curl -s https://api.github.com/repos/prometheus-operator/prometheus-operator/releases/latest | jq -cr .tag_name) curl https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/$LATEST/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml | kubectl create -f - ``` ### Using Manifests If you're using *Kubernetes manifests* (Deployment, DaemonSet, or StatefulSet) to install the Ingress Controller, to enable Prometheus metrics: 1. Run the Ingress Controller with the `-enable-prometheus-metrics` [command-line argument](/nic/lts/configuration/global-configuration/command-line-arguments.md). As a result, the Ingress Controller will expose NGINX Plus metrics in the Prometheus format via the path `/metrics` on port `9113` (customizable via the `-prometheus-metrics-listen-port` command-line argument). 1. To enable TLS for the Prometheus endpoint, configure the `-prometheus-tls-secret` cli argument with the namespace and name of a TLS Secret. 1. Add the Prometheus port to the list of the ports of the Ingress Controller container in the template of the Ingress Controller pod: ```yaml - name: prometheus containerPort: 9113 ``` 1. Make Prometheus aware of the Ingress Controller targets by adding the following annotations to the template of the Ingress Controller pod (note: this assumes your Prometheus is configured to discover targets by analyzing the annotations of pods): ```yaml annotations: prometheus.io/scrape: "true" prometheus.io/port: "9113" prometheus.io/scheme: http ``` ## Available Metrics The Ingress Controller exports the following metrics: - NGINX Plus metrics: - Exported by NGINX Plus. Refer to the [NGINX Prometheus Exporter developer docs](https://github.com/nginx/nginx-prometheus-exporter#exported-metrics) to find more information about the exported metrics. - There is a Grafana dashboard for NGINX Plus metrics located in the root repo folder. - Calculated by the Ingress Controller: - `controller_upstream_server_response_latency_ms_count`. Bucketed response times from when NGINX establishes a connection to an upstream server to when the last byte of the response body is received by NGINX. **Note**: The metric for the upstream isn't available until traffic is sent to the upstream. The metric isn't enabled by default. To enable the metric, set the `-enable-latency-metrics` command-line argument. - Ingress Controller metrics - `controller_nginx_reloads_total`. Number of successful NGINX reloads. This includes the label `reason` with 2 possible values `endpoints` (the reason for the reload was an endpoints update) and `other` (the reload was caused by something other than an endpoint update like an ingress update). - `controller_nginx_reload_errors_total`. Number of unsuccessful NGINX reloads. - `controller_nginx_last_reload_status`. Status of the last NGINX reload, 0 meaning down and 1 up. - `controller_nginx_last_reload_milliseconds`. Duration in milliseconds of the last NGINX reload. - `controller_nginx_worker_processes_total`. Number of NGINX worker processes. This metric includes the constant label `generation` with two possible values `old` (the shutting down processes of the old generations) or `current` (the processes of the current generation). - `controller_ingress_resources_total`. Number of handled Ingress resources. This metric includes the label type, that groups the Ingress resources by their type (regular, [minion or master](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md). **Note**: The metric doesn't count minions without a master. - `controller_virtualserver_resources_total`. Number of handled VirtualServer resources. - `controller_virtualserverroute_resources_total`. Number of handled VirtualServerRoute resources. **Note**: The metric counts only VirtualServerRoutes that have a reference from a VirtualServer. - `location_zone` (upstream services) metrics: - `location_zone_sent`. Number of bytes sent to clients. - `location_zone_received`. Number of bytes received from clients. - `location_zone_requests`. Total number of client requests. - `location_zone_responses`. Total number of responses sent to clients. - `location_zone_responses_codes`. Total number of responses sent to clients. - `location_zone_sent`. Number of bytes sent to clients. - `controller_transportserver_resources_total`. Number of handled TransportServer resources. This metric includes the label type, that groups the TransportServer resources by their type (passthrough, tcp or udp). - Workqueue metrics. **Note**: the workqueue is a queue used by the Ingress Controller to process changes to the relevant resources in the cluster like Ingress resources. The Ingress Controller uses only one queue. The metrics for that queue will have the label `name="taskQueue"` - `workqueue_depth`. Current depth of the workqueue. - `workqueue_queue_duration_second`. How long in seconds an item stays in the workqueue before being requested. - `workqueue_work_duration_seconds`. How long in seconds processing an item from the workqueue takes. **Note**: all metrics have the namespace `nginx_ingress`. For example, `nginx_ingress_controller_nginx_reloads_total`. **Note**: all metrics include the label `class`, which is set to the class of the Ingress Controller. The class is configured via the `-ingress-class` command-line argument. -------------------------------------------------------------------------------- title: "F5 BIG-IP" url: https://docs.nginx.com/nginx-ingress-controller/integrations/f5-ingresslink/index.md -------------------------------------------------------------------------------- Learn how to use F5 IngressLink with NGINX Ingress Controller to configure your F5 BIG-IP device. F5 IngressLink is an integration between NGINX Ingress Controller and [F5 BIG-IP Container Ingress Services](https://clouddocs.f5.com/containers/latest/) (CIS) that configures an F5 BIG-IP device as a load balancer for NGINX Ingress Controller pods. ## Install NGINX Ingress Controller with the integration enabled The steps to enable the integration depend on the option chosen to install NGINX Ingress Controller: Using [Manifests](/nic/install/manifests) or using the [Helm chart](/nic/install/helm). ### Installation using manifests 1. Create a service for the Ingress Controller pods for ports 80 and 443. For example: ```yaml apiVersion: v1 kind: Service metadata: name: nginx-ingress-ingresslink namespace: nginx-ingress labels: app: ingresslink spec: ports: - port: 80 targetPort: 80 protocol: TCP name: http - port: 443 targetPort: 443 protocol: TCP name: https selector: app: nginx-ingress ``` Note the label `app: ingresslink`. We will use it in the [Configure CIS](#configure-cis) step. 1. In the [ConfigMap resource](/nic/configuration/global-configuration/configmap-resource) enable the proxy protocol, which the BIG-IP system will use to pass the client IP and port information to NGINX. For the `set-real-ip-from` key, use the subnet of the IP which the BIG-IP system uses to send traffic to NGINX: ```yaml proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" ``` 1. Deploy NGINX Ingress Controller with additional [command-line arguments](/nic/configuration/global-configuration/command-line-arguments): ```yaml args: - -ingresslink=nginx-ingress - -report-ingress-status . . . ``` where `ingresslink` references the name of the IngressLink resource from step 1, and `report-ingress-status` enables [reporting ingress statuses](/nic/configuration/global-configuration/reporting-resources-status#ingress-resources). ### Installation using Helm Install a Helm release with the following values: ```yaml controller: config: entries: proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" reportIngressStatus: ingressLink: nginx-ingress service: type: ClusterIP externalTrafficPolicy: Cluster extraLabels: app: ingresslink ``` We will use the `ingressLink` and `extraLabels` parameter values to configure CIS in the next section. For the `set-real-ip-from` key, use the subnet of the IP which the BIG-IP system uses to send traffic to NGINX. ## Configure CIS To enable the integration, F5 BIG-IP Container Ingress Services must be deployed in the cluster and configured to support the integration. Follow the instructions on the [CIS documentation portal](https://clouddocs.f5.com/containers/latest/userguide/ingresslink/#configuring-ingresslink). Make sure that: - The name of the IngressLink resource is the same as the one used during the installation of NGINX Ingress Controller (`nginx-ingress` in the previous example). - The selector in the IngressLink resource is the same as the Service labels configured during Ingress Controller installation (`app: ingresslink` in the previous example). - The IngressLink must belong to the same namespace as the Ingress Controller pod (`nginx-ingress` or the namespace used for installing the Helm chart). -------------------------------------------------------------------------------- title: "F5 BIG-IP" url: https://docs.nginx.com/nginx-ingress-controller/lts/integrations/f5-ingresslink/index.md -------------------------------------------------------------------------------- Learn how to use F5 IngressLink with NGINX Ingress Controller LTS to configure your F5 BIG-IP device. F5 IngressLink is an integration between NGINX Ingress Controller LTS and [F5 BIG-IP Container Ingress Services](https://clouddocs.f5.com/containers/latest/) (CIS) that configures an F5 BIG-IP device as a load balancer for NGINX Ingress Controller LTS pods. ## Install NGINX Ingress Controller LTS with the integration enabled The steps to enable the integration depend on the option chosen to install NGINX Ingress Controller LTS: Using [Manifests](/nic/lts/install/manifests) or using the [Helm chart](/nic/lts/install/helm). ### Installation using manifests 1. Create a service for the Ingress Controller pods for ports 80 and 443. For example: ```yaml apiVersion: v1 kind: Service metadata: name: nginx-ingress-ingresslink namespace: nginx-ingress labels: app: ingresslink spec: ports: - port: 80 targetPort: 80 protocol: TCP name: http - port: 443 targetPort: 443 protocol: TCP name: https selector: app: nginx-ingress ``` Note the label `app: ingresslink`. We will use it in the [Configure CIS](#configure-cis) step. 1. In the [ConfigMap resource](/nic/lts/configuration/global-configuration/configmap-resource) enable the proxy protocol, which the BIG-IP system will use to pass the client IP and port information to NGINX. For the `set-real-ip-from` key, use the subnet of the IP which the BIG-IP system uses to send traffic to NGINX: ```yaml proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" ``` 1. Deploy NGINX Ingress Controller LTS with additional [command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments): ```yaml args: - -ingresslink=nginx-ingress - -report-ingress-status . . . ``` where `ingresslink` references the name of the IngressLink resource from step 1, and `report-ingress-status` enables [reporting ingress statuses](/nic/lts/configuration/global-configuration/reporting-resources-status#ingress-resources). ### Installation using Helm Install a Helm release with the following values: ```yaml controller: config: entries: proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" reportIngressStatus: ingressLink: nginx-ingress service: type: ClusterIP externalTrafficPolicy: Cluster extraLabels: app: ingresslink ``` We will use the `ingressLink` and `extraLabels` parameter values to configure CIS in the next section. For the `set-real-ip-from` key, use the subnet of the IP which the BIG-IP system uses to send traffic to NGINX. ## Configure CIS To enable the integration, F5 BIG-IP Container Ingress Services must be deployed in the cluster and configured to support the integration. Follow the instructions on the [CIS documentation portal](https://clouddocs.f5.com/containers/latest/userguide/ingresslink/#configuring-ingresslink). Make sure that: - The name of the IngressLink resource is the same as the one used during the installation of NGINX Ingress Controller LTS (`nginx-ingress` in the previous example). - The selector in the IngressLink resource is the same as the Service labels configured during Ingress Controller installation (`app: ingresslink` in the previous example). - The IngressLink must belong to the same namespace as the Ingress Controller pod (`nginx-ingress` or the namespace used for installing the Helm chart). -------------------------------------------------------------------------------- title: "NGINX Ingress Controller Helm chart parameters" url: https://docs.nginx.com/nginx-ingress-controller/install/helm/parameters/index.md -------------------------------------------------------------------------------- This page describes the Helm chart parameters of F5 NGINX Ingress Controllers. It is a reference for each parameter name, a description of the parameter, and any default values. ## NGINX Ingress Controller Helm chart parameters **Note:** The [values.yaml](https://github.com/nginx/kubernetes-ingress/blob/main/charts/nginx-ingress/values.yaml) file of the Helm chart is a single source of truth and includes parameter descriptions: the table may become outdated. The [values.schema.json](https://github.com/nginx/kubernetes-ingress/blob/main/charts/nginx-ingress/values.schema.json) file can be used to inspect how values are formatted. | Parameter | Description | Default | | ----------| ----------- | ------- | | **controller.name** | The name of the NGINX Ingress Controller deployment, daemonset, or statefulset. | Autogenerated | | **controller.kind** | The kind of the NGINX Ingress Controller installation - deployment, daemonset, or statefulset. | deployment | | **controller.annotations** | Allows for setting of `annotations` for deployment, daemonset, or statefulset. | {} | | **controller.nginxplus** | Deploys NGINX Ingress Controller for NGINX Plus. | false | | **controller.mgmt.licenseTokenSecretName** | Configures the secret used in the [license_token](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.com/license` with the base64 encoded JWT in the `license.jwt` key. | license-token | | **controller.mgmt.enforceInitialReport** | Configures the [enforce_initial_report](https://nginx.org/en/docs/ngx_mgmt_module.html#enforce_initial_report) directive, which enables or disables the 180-day grace period for sending the initial usage report. | false | | **controller.mgmt.usageReport.endpoint** | Configures the endpoint of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This is used to configure the endpoint NGINX uses to send usage reports to NGINX Instance Manager . | product.connect.nginx.com | | **controller.mgmt.usageReport.interval** | Configures the interval of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This specifies the frequency that usage reports are sent. Only seconds(s), minutes(m), and hours(h) are allowed and must be between 60s and 24h. | 1h | | **controller.mgmt.usageReport.proxyHost** | Configures the host name of the [proxy](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy) directive with optional port. | N/A | | **controller.mgmt.usageReport.proxyCredentialsSecretName** | Configures the [proxy_username](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy_username) directive as well as the [proxy_password](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy_password) directive using a Kubernetes Opaque Secret. The Secret must contain `username` and `password` fields. | N/A | | **controller.mgmt.sslVerify** | Configures the [ssl_verify](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_verify) directive, which enables or disables verification of the usage reporting endpoint certificate. | true | | **controller.mgmt.resolver.ipv6** | Configures whether the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will look up IPv6 addresses. | true | | **controller.mgmt.resolver.valid** | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will override the TTL value of responses from nameservers with. | N/A | | **controller.mgmt.resolver.addresses** | Configures addresses used in the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive. This field takes a list of addresses. | N/A | | **controller.mgmt.sslCertificateSecretName** | Configures the secret used to create the `ssl_certificate` and `ssl_certificate_key` directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `kubernetes.io/tls` | N/A | | **controller.mgmt.sslTrustedCertificateSecretName** | Configures the secret used to create the file(s) referenced the in [ssl_trusted_certifcate](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_trusted_certificate), and [ssl_crl](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller is deployed in. The secret must be of type `nginx.org/ca`, where the `ca.crt` key contains a base64 encoded trusted cert, and the optional `ca.crl` key can contain a base64 encoded CRL. If the optional `ca.crl` key is supplied, it will configure the NGINX `ssl_crl` directive. | N/A | | **controller.mgmt.configMapName** | Allows changing the name of the MGMT config map. The name should not include a namespace| Autogenerated | | **controller.nginxReloadTimeout** | The timeout in milliseconds which the NGINX Ingress Controller will wait for a successful NGINX reload after a change or at the initial start. | 60000 | | **controller.hostNetwork** | Enables NGINX Ingress Controller pods to use the host's network namespace. | false | | **controller.dnsPolicy** | DNS policy for NGINX Ingress Controller pods. | ClusterFirst | | **controller.nginxDebug** | Enables debugging for NGINX. Uses the `nginx-debug` binary. Requires `error-log-level: debug` in the ConfigMap via `controller.config.entries`. | false | | **controller.enableConfigSafety** | Enhances the stability and reliability of the NGINX Ingress Controller. When this feature is turned on, it ensures the validation of new configurations before reloading nginx. If a configuration is deemed invalid, the new config for the associated resource is rejected and the old working config is restored. This is an experimental feature. Behavior and configuration may change in future releases. When turned on, users can experience delayed pod startup times as resources are validated prior to being written. Delays are directly proportional to the number and complexity of resources. | false | | **controller.logLevel** | The log level of NGINX Ingress Controller. | info | | **controller.logFormat** | The log format of NGINX Ingress Controller. | glog | | **controller.directiveAutoAdjust** | Automatically adjusts NGINX buffer directives to prevent configuration errors. | false | | **controller.image.digest** | The image digest of NGINX Ingress Controller. | None | | **controller.image.repository** | The image repository of NGINX Ingress Controller. | nginx/nginx-ingress | | **controller.image.tag** | The tag of the NGINX Ingress Controller image. | | | **controller.image.pullPolicy** | The pull policy for the NGINX Ingress Controller image. | IfNotPresent | | **controller.lifecycle** | The lifecycle of the NGINX Ingress Controller pods. | {} | | **controller.customConfigMap** | The name of the custom ConfigMap used by the NGINX Ingress Controller. If set, then the default config is ignored. | "" | | **controller.config.name** | The name of the ConfigMap used by the NGINX Ingress Controller. | Autogenerated | | **controller.config.annotations** | The annotations of the NGINX Ingress Controller configmap. | {} | | **controller.config.entries** | The entries of the ConfigMap for customizing NGINX configuration. See [ConfigMap resource docs](/nic/configuration/global-configuration/configmap-resource.md) for the list of supported ConfigMap keys. | {} | | **controller.customPorts** | A list of custom ports to expose on the NGINX Ingress Controller pod. Follows the conventional Kubernetes yaml syntax for container ports. | [] | | **controller.defaultTLS.cert** | The base64-encoded TLS certificate for the default HTTPS server. **Note:** It is recommended that you specify your own certificate. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server. | | **controller.defaultTLS.key** | The base64-encoded TLS key for the default HTTPS server. **Note:** It is recommended that you specify your own key. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server. | | **controller.defaultTLS.secret** | The secret with a TLS certificate and key for the default HTTPS server. The value must follow the following format: `/`. Used as an alternative to specifying a certificate and key using `controller.defaultTLS.cert` and `controller.defaultTLS.key` parameters. **Note:** Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server. When `controller.allowEmptyIngressHost` is enabled, this secret also provides TLS for Ingress resources without a `host`. | None | | **controller.wildcardTLS.cert** | The base64-encoded TLS certificate for every Ingress/VirtualServer host that has TLS enabled but no secret specified. If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection. | None | | **controller.wildcardTLS.key** | The base64-encoded TLS key for every Ingress/VirtualServer host that has TLS enabled but no secret specified. If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection. | None | | **controller.wildcardTLS.secret** | The secret with a TLS certificate and key for every Ingress/VirtualServer host that has TLS enabled but no secret specified. The value must follow the following format: `/`. Used as an alternative to specifying a certificate and key using `controller.wildcardTLS.cert` and `controller.wildcardTLS.key` parameters. | None | | **controller.nodeSelector** | The node selector for pod assignment for NGINX Ingress Controller pods. | {} | | **controller.terminationGracePeriodSeconds** | The termination grace period of the NGINX Ingress Controller pod. | 30 | | **controller.tolerations** | The tolerations of NGINX Ingress Controller pods. | [] | | **controller.affinity** | The affinity of NGINX Ingress Controller pods. | {} | | **controller.topologySpreadConstraints** | The topology spread constraints of the NGINX Ingress Controller pods. | {} | | **controller.env** | The additional environment variables to be set on the NGINX Ingress Controller pods. | [] | | **controller.volumes** | The volumes of NGINX Ingress Controller pods. | [] | | **controller.volumeMounts** | The volumeMounts of NGINX Ingress Controller pods. | [] | | **controller.initContainers** | InitContainers for NGINX Ingress Controller pods. | [] | | **controller.extraContainers** | Extra (eg. sidecar) containers for NGINX Ingress Controller pods. | [] | | **controller.podSecurityContext**| The SecurityContext for Ingress Controller pods. | "seccompProfile": {"type": "RuntimeDefault"} | | **controller.securityContext** | The SecurityContext for Ingress Controller container. | {} | | **controller.initContainerSecurityContext** | The SecurityContext for NGINX Ingress Controller init container when `readOnlyRootFilesystem` is enabled by either setting `controller.securityContext.readOnlyRootFilesystem` or `controller.readOnlyRootFilesystem`to `true`. | {} | | **controller.resources** | The resources of NGINX Ingress Controller pods. | requests: cpu=100m,memory=128Mi | | **controller.initContainerResources** | The resources of the init container which is used when `readOnlyRootFilesystem` is enabled by either setting `controller.securityContext.readOnlyRootFilesystem` or `controller.readOnlyRootFilesystem`to `true`. | requests: cpu=100m,memory=128Mi | | **controller.replicaCount** | The number of replicas of the NGINX Ingress Controller deployment. | 1 | | **controller.ingressClass.name** | A class of NGINX Ingress Controller. An IngressClass resource with the name equal to the class must be deployed. Otherwise, the NGINX Ingress Controller will fail to start. the NGINX Ingress Controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class. the NGINX Ingress Controller processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field for all versions of Kubernetes. | nginx | | **controller.ingressClass.create** | Creates a new IngressClass object with the name `controller.ingressClass.name`. Set to `false` to use an existing ingressClass created using `kubectl` with the same name. If you use `helm upgrade`, do not change the values from the previous release as helm will delete IngressClass objects managed by helm. If you are upgrading from a release earlier than , do not set the value to false. | true | | **controller.ingressClass.setAsDefaultIngress** | New Ingresses without an `"ingressClassName"` field specified will be assigned the class specified in `controller.ingressClass.name`. Requires `controller.ingressClass.create`. | false | | **controller.watchNamespace** | Comma separated list of namespaces the NGINX Ingress Controller should watch for resources. By default the NGINX Ingress Controller watches all namespaces. Mutually exclusive with `controller.watchNamespaceLabel`. Please note that if configuring multiple namespaces using the Helm cli `--set` option, the string needs to wrapped in double quotes and the commas escaped using a backslash - e.g. `--set controller.watchNamespace="default\,nginx-ingress"`. | "" | | **controller.watchNamespaceLabel** | Configures NGINX Ingress Controller to watch only those namespaces with label foo=bar. By default the NGINX Ingress Controller watches all namespaces. Mutually exclusive with `controller.watchNamespace`. | "" | | **controller.watchSecretNamespace** | Comma separated list of namespaces the NGINX Ingress Controller should watch for resources of type Secret. If this arg is not configured, the NGINX Ingress Controller watches the same namespaces for all resources, see `controller.watchNamespace` and `controller.watchNamespaceLabel`. All namespaces included with this argument must be part of either `controller.watchNamespace` or `controller.watchNamespaceLabel`. Please note that if configuring multiple namespaces using the Helm cli `--set` option, the string needs to wrapped in double quotes and the commas escaped using a backslash - e.g. `--set controller.watchSecretNamespace="default\,nginx-ingress"`. | "" | | **controller.enableCustomResources** | Enable the custom resources. | true | | **controller.enableOIDC** | Enable OIDC policies. | false | | **controller.enableTLSPassthrough** | Enable TLS Passthrough on default port 443. Requires `controller.enableCustomResources`. | false | | **controller.tlsPassThroughPort** | Set the port for the TLS Passthrough. Requires `controller.enableCustomResources` and `controller.enableTLSPassthrough`. | 443 | | **controller.enableCertManager** | Enable x509 automated certificate management for VirtualServer resources using cert-manager (cert-manager.io). Requires `controller.enableCustomResources`. | false | | **controller.enableExternalDNS** | Enable integration with ExternalDNS for configuring public DNS entries for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Requires `controller.enableCustomResources`. | false | | **controller.globalConfiguration.create** | Creates the GlobalConfiguration custom resource. Requires `controller.enableCustomResources`. | false | | **controller.globalConfiguration.spec** | The spec of the GlobalConfiguration for defining the global configuration parameters of the NGINX Ingress Controller. | {} | | **controller.enableSnippets** | Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources. | false | | **controller.healthStatus** | Add a location "/nginx-health" to the default server. The location responds with the 200 status code for any request. Useful for external health-checking of the NGINX Ingress Controller. | false | | **controller.healthStatusURI** | Sets the URI of health status location in the default server. Requires `controller.healthStatus`. | "/nginx-health" | | **controller.nginxStatus.enable** | Enable the NGINX stub_status, or the NGINX Plus API. | true | | **controller.nginxStatus.port** | Set the port where the NGINX stub_status or the NGINX Plus API is exposed. | 8080 | | **controller.nginxStatus.allowCidrs** | Add IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas. | 127.0.0.1,::1 | | **controller.priorityClassName** | The PriorityClass of NGINX Ingress Controller pods. | None | | **controller.service.create** | Creates a service to expose NGINX Ingress Controller pods. | true | | **controller.service.type** | The type of service to create for NGINX Ingress Controller. | LoadBalancer | | **controller.service.externalTrafficPolicy** | The externalTrafficPolicy of the service. The value Local preserves the client source IP. | Local | | **controller.service.annotations** | The annotations of the NGINX Ingress Controller service. | {} | | **controller.service.extraLabels** | The extra labels of the service. | {} | | **controller.service.loadBalancerIP** | The static IP address for the load balancer. Requires `controller.service.type` set to `LoadBalancer`. The cloud provider must support this feature. | "" | | **controller.service.loadBalancerClass** | This property allows the user to define a specific load balancer implementation or configuration class for the service. Requires `controller.service.type` set to `LoadBalancer`. The cloud provider must support this feature. | "" | | **controller.service.externalIPs** | The list of external IPs for the NGINX Ingress Controller service. | [] | | **controller.service.clusterIP** | The clusterIP for the NGINX Ingress Controller service, autoassigned if not specified. | "" | | **controller.service.loadBalancerSourceRanges** | The IP ranges (CIDR) that are allowed to access the load balancer. Requires `controller.service.type` set to `LoadBalancer`. The cloud provider must support this feature. | [] | | **controller.service.name** | The name of the service. | Autogenerated | | **controller.service.customPorts** | A list of custom ports to expose through the NGINX Ingress Controller service. Follows the conventional Kubernetes yaml syntax for service ports. | [] | | **controller.service.httpPort.enable** | Enables the HTTP port for the NGINX Ingress Controller service. | true | | **controller.service.httpPort.port** | The HTTP port of the NGINX Ingress Controller service. | 80 | | **controller.service.httpPort.nodePort** | The custom NodePort for the HTTP port. Requires `controller.service.type` set to `NodePort`. | "" | | **controller.service.httpPort.targetPort** | The target port of the HTTP port of the NGINX Ingress Controller service. | 80 | | **controller.service.httpsPort.enable** | Enables the HTTPS port for the NGINX Ingress Controller service. | true | | **controller.service.httpsPort.port** | The HTTPS port of the NGINX Ingress Controller service. | 443 | | **controller.service.httpsPort.nodePort** | The custom NodePort for the HTTPS port. Requires `controller.service.type` set to `NodePort`. | "" | | **controller.service.httpsPort.targetPort** | The target port of the HTTPS port of the NGINX Ingress Controller service. | 443 | | **controller.service.sessionAffinity.enable** | Enable session affinity. | `false` | | **controller.service.sessionAffinity.type** | Session affinity type. Valid values: `None`, `ClientIP`. | `ClientIP` | | **controller.service.sessionAffinity.timeout** | Session affinity timeout in seconds. | 3600 | | **controller.serviceAccount.annotations** | The annotations of the NGINX Ingress Controller service account. | {} | | **controller.serviceAccount.name** | The name of the service account of the NGINX Ingress Controller pods. Used for RBAC. | Autogenerated | | **controller.serviceAccount.imagePullSecretName** | The name of the secret containing docker registry credentials. Secret must exist in the same namespace as the helm release. | "" | | **controller.serviceAccount.imagePullSecretsNames** | The list of secret names containing docker registry credentials. Secret must exist in the same namespace as the helm release. | [] | | **controller.reportIngressStatus.enable** | Updates the address field in the status of Ingress resources with an external address of the NGINX Ingress Controller. You must also specify the source of the external address either through an external service via `controller.reportIngressStatus.externalService`, `controller.reportIngressStatus.ingressLink` or the `external-status-address` entry in the ConfigMap via `controller.config.entries`. **Note:** `controller.config.entries.external-status-address` takes precedence over the others. | true | | **controller.reportIngressStatus.externalService** | Specifies the name of the service with the type LoadBalancer through which the NGINX Ingress Controller is exposed externally. The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. `controller.reportIngressStatus.enable` must be set to `true`. The default is autogenerated and enabled when `controller.service.create` is set to `true` and `controller.service.type` is set to `LoadBalancer`. | Autogenerated | | **controller.reportIngressStatus.ingressLink** | Specifies the name of the IngressLink resource, which exposes the NGINX Ingress Controller pods via a BIG-IP system. The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. `controller.reportIngressStatus.enable` must be set to `true`. | "" | | **controller.reportIngressStatus.enableLeaderElection** | Enable Leader election to avoid multiple replicas of the controller reporting the status of Ingress resources. `controller.reportIngressStatus.enable` must be set to `true`. | true | | **controller.reportIngressStatus.leaderElectionLockName** | Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. controller.reportIngressStatus.enableLeaderElection must be set to true. | Autogenerated | | **controller.reportIngressStatus.annotations** | The annotations of the leader election configmap. | {} | | **controller.pod.annotations** | The annotations of the NGINX Ingress Controller pod. | {} | | **controller.pod.extraLabels** | The additional extra labels of the NGINX Ingress Controller pod. | {} | | **controller.appprotect.enable** | Enables the F5 WAF for NGINX module in the NGINX Ingress Controller. | false | | **controller.appprotect.v5** | Enables F5 WAF for NGINX v5. | false | | **controller.appprotect.plmStorage.url** | SeaweedFS S3 endpoint from which NGINX Ingress Controller fetches the policy and logconf bundle. Leave empty to turn off PLM support. Requires `controller.appprotect.v5` to be `true`. | "" | | **controller.appprotect.plmStorage.credentialsSecret** | Secret containing the SeaweedFS admin secret in the `seaweedfs_admin_secret` key. Format: `/`. | "" | | **controller.appprotect.plmStorage.caSecret** | Optional Secret containing the SeaweedFS admin secret under `seaweedfs_admin_secret`. Format: `/`. | "" | | **controller.appprotect.plmStorage.clientSSLSecret** | Optional Secret containing `tls.crt` and `tls.key` for SeaweedFS mTLS. Format: `/`. | "" | | **controller.appprotect.plmStorage.insecureSkipVerify** | Turns off SeaweedFS TLS verification. For development and testing only. | false | | **controller.appprotect.volumes** | Volumes for F5 WAF for NGINX v5. | [{"name": "app-protect-bd-config", "emptyDir": {}},{"name": "app-protect-config", "emptyDir": {}},{"name": "app-protect-bundles", "emptyDir": {}}] | | **controller.appprotect.enforcer.host** | Host that the F5 WAF for NGINX v5 Enforcer runs on. | "127.0.0.1" | | **controller.appprotect.enforcer.port** | Port that the F5 WAF for NGINX v5 Enforcer runs on. | 50000 | | **controller.appprotect.enforcer.image.repository** | The image repository of the F5 WAF for NGINX v5 Enforcer. | private-registry.nginx.com/nap/waf-enforcer | | **controller.appprotect.enforcer.image.tag** | The tag of the F5 WAF for NGINX v5 Enforcer. | "5.6.0" | | **controller.appprotect.enforcer.image.digest** | The digest of the F5 WAF for NGINX v5 Enforcer. Takes precedence over tag if set. | "" | | **controller.appprotect.enforcer.image.pullPolicy** | The pull policy for the F5 WAF for NGINX v5 Enforcer image. | IfNotPresent | | **controller.appprotect.enforcer.securityContext** | The security context for F5 WAF for NGINX v5 Enforcer container. | {} | | **controller.appprotect.configManager.image.repository** | The image repository of the F5 WAF for NGINX v5 Configuration Manager. | private-registry.nginx.com/nap/waf-config-mgr | | **controller.appprotect.configManager.image.tag** | The tag of the F5 WAF for NGINX v5 Configuration Manager. | "5.6.0" | | **controller.appprotect.configManager.image.digest** | The digest of the F5 WAF for NGINX v5 Configuration Manager. Takes precedence over tag if set. | "" | | **controller.appprotect.configManager.image.pullPolicy** | The pull policy for the F5 WAF for NGINX v5 Configuration Manager image. | IfNotPresent | | **controller.appprotect.configManager.securityContext** | The security context for F5 WAF for NGINX v5 Configuration Manager container. | {"allowPrivilegeEscalation":false,"runAsUser":101,"runAsNonRoot":true,"capabilities":{"drop":["all"]}} | | **controller.appprotect.logLevel** | Sets the log level for F5 WAF for NGINX | N/A | | **controller.appprotectdos.enable** | Enables the App Protect DoS module in the NGINX Ingress Controller. | false | | **controller.appprotectdos.debug** | Enable debugging for App Protect DoS. | false | | **controller.appprotectdos.maxDaemons** | Max number of ADMD instances. | 1 | | **controller.appprotectdos.maxWorkers** | Max number of nginx processes to support. | Number of CPU cores in the machine | | **controller.appprotectdos.memory** | RAM memory size to consume in MB. | 50% of free RAM in the container or 80MB, the smaller | | **controller.readyStatus.enable** | Enables the readiness endpoint `"/nginx-ready"`. The endpoint returns a success code when NGINX has loaded all the config after the startup. This also configures a readiness probe for the NGINX Ingress Controller pods that uses the readiness endpoint. | true | | **controller.readyStatus.port** | The HTTP port for the readiness endpoint. | 8081 | | **controller.readyStatus.initialDelaySeconds** | The number of seconds after the NGINX Ingress Controller pod has started before readiness probes are initiated. | 0 | | **controller.startupStatus.enable** | Enables the startup probe for the Ingress Controller. | false | | **controller.startupStatus.port** | The port where the startup endpoint is exposed. This is a required field if `controller.startupStatus.enable` is set to true. | N/A | | **controller.startupStatus.path** | The path to the startup endpoint. This is a required field if `controller.startupStatus.enable` is set to true. | N/A | | **controller.startupStatus.initialDelaySeconds** | The number of seconds after the Ingress Controller pod has started before startup probes are initiated. | N/A | | **controller.startupStatus.periodSeconds** | The number of seconds between each startup probe. | N/A | | **controller.startupStatus.timeoutSeconds** | The number of seconds after which the startup probe times out. | N/A | | **controller.startupStatus.successThreshold** | Minimum consecutive successes for the probe to be considered successful. | N/A | | **controller.startupStatus.failureThreshold** | When a probe fails, Kubernetes will try failureThreshold times before giving up. | N/A | | **controller.enableLatencyMetrics** | Enable collection of latency metrics for upstreams. Requires `prometheus.create`. | false | | **controller.minReadySeconds** | Specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. [docs](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#min-ready-seconds) | 0 | | **controller.autoscaling.enabled** | Enables HorizontalPodAutoscaling. | false | | **controller.autoscaling.create** | Create the HorizontalPodAutoscaler resource. This can be set to false to manage the HPA externally. | true | | **controller.autoscaling.annotations** | The annotations of the NGINX Ingress Controller HorizontalPodAutoscaler. | {} | | **controller.autoscaling.behavior** | Behavior configuration for the HPA. | {} | | **controller.autoscaling.minReplicas** | Minimum number of replicas for the HPA. | 1 | | **controller.autoscaling.maxReplicas** | Maximum number of replicas for the HPA. | 3 | | **controller.autoscaling.targetCPUUtilizationPercentage** | The target CPU utilization percentage. | 50 | | **controller.autoscaling.targetMemoryUtilizationPercentage** | The target memory utilization percentage. | 50 | | **controller.podDisruptionBudget.enabled** | Enables PodDisruptionBudget. | false | | **controller.podDisruptionBudget.annotations** | The annotations of the NGINX Ingress Controller pod disruption budget | {} | | **controller.podDisruptionBudget.minAvailable** | The number of Ingress Controller pods that should be available. This is a mutually exclusive setting with "maxUnavailable". | 0 | | **controller.podDisruptionBudget.maxUnavailable** | The number of Ingress Controller pods that can be unavailable. This is a mutually exclusive setting with "minAvailable". | 0 | | **controller.strategy** | Specifies the strategy used to replace old Pods with new ones. Docs for [Deployment update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy), [Daemonset update strategy](https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/#daemonset-update-strategy) and [StatefulSet update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies) | {} | | **controller.statefulset.podManagementPolicy** | Controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. Available options: `OrderedReady` or `Parallel`. Only applies when `controller.kind` is set to `statefulset`. | `OrderedReady` | | **controller.statefulset.persistentVolumeClaimRetentionPolicy.whenDeleted** | Controls the retention policy for PVCs when the StatefulSet is deleted. Available options: `Retain` or `Delete`. Only applies when `controller.kind` is set to `statefulset`. | `Retain` | | **controller.statefulset.persistentVolumeClaimRetentionPolicy.whenScaled** | Controls the retention policy for PVCs when the StatefulSet is scaled down. Available options: `Retain` or `Delete`. Only applies when `controller.kind` is set to `statefulset`. | `Retain` | | **controller.statefulset.nginxCachePVC.size** | The size of the persistent volume claim for NGINX cache storage. Only applies when `controller.kind` is set to `statefulset`. | `256Mi` | | **controller.statefulset.nginxCachePVC.storageClass** | The storage class for the persistent volume claim. Only applies when `controller.kind` is set to `statefulset`. | `""` | | **controller.statefulset.nginxCachePVC.accessModes** | The access modes for the persistent volume claim. Only applies when `controller.kind` is set to `statefulset`. | `["ReadWriteOnce"]` | | **controller.disableIPV6** | Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack. | false | | **controller.defaultHTTPListenerPort** | Sets the port for the HTTP `default_server` listener. When `controller.allowEmptyIngressHost` is enabled, also configures the HTTP listener port for Ingress resources without a `host`. | 80 | | **controller.defaultHTTPSListenerPort** | Sets the port for the HTTPS `default_server` listener. When `controller.allowEmptyIngressHost` is enabled, also configures the HTTPS listener port for Ingress resources without a `host`. | 443 | | **controller.allowEmptyIngressHost** | Allows Ingress resources to omit the `host` field. `spec.tls` is not supported on Ingress resources without a `host`; use `controller.defaultTLS.secret` instead. `nginx.org/listen-ports` and `nginx.org/listen-ports-ssl` annotations are not supported; use `controller.defaultHTTPListenerPort` and `controller.defaultHTTPSListenerPort` instead. | false | | **controller.readOnlyRootFilesystem** | Configure root filesystem as read-only and add volumes for temporary data. Three major releases after 3.5.x this argument will be moved permanently to the `controller.securityContext` section. | false | | **controller.enableSSLDynamicReload** | Enable lazy loading for SSL Certificates. | true | | **controller.telemetryReporting.enable** | Enable telemetry reporting. | true | | **controller.enableWeightChangesDynamicReload** | Enable weight changes without reloading the NGINX configuration. May require increasing `map_hash_bucket_size`, `map_hash_max_size`, `variable_hash_bucket_size`, and `variable_hash_max_size` in the [ConfigMap](/nic/configuration/global-configuration/configmap-resource.md) if there are many two-way splits. Requires `controller.nginxplus` | false | | **controller.networkPolicy.enabled** | Create a NetworkPolicy that targets the controller pods. If `enabled` is set to `true` and both ingress and egress are empty, the resulting NetworkPolicy denies all ingress by default. | false | | **controller.networkPolicy.ingress** | Configure ingress rules. When non-empty, the Ingress policy type is added automatically. | [] | | **controller.networkPolicy.egress** | Configure egress rules. When non-empty, the Egress policy type is added automatically. | [] | |**nginxAgent.enable** | Enable NGINX Agent 3.x to allow [connecting to NGINX One Console](/nginx-one-console/k8s/add-nic.md) or to integrate NGINX Agent 2.x for [Security Monitoring](/nic/tutorials/security-monitoring.md) . | false | |**nginxAgent.logLevel** | Log level for NGINX Agent. | "error" | |**nginxAgent.dataplaneKeySecretName** | Name of the Kubernetes Secret containing the Data Plane key used to authenticate to NGINX One Console. Learn more [here](/nginx-one-console/k8s/add-nic.md). Required when `nginxAgent.enable` is set to `true`. Requires NGINX Agent 3.x. | "" | |**nginxAgent.endpointHost** | Domain or IP address for the NGINX One Console. Requires NGINX Agent 3.x. | "agent.connect.nginx.com" | |**nginxAgent.endpointPort** | Port for the NGINX One Console endpoint. Requires NGINX Agent 3.x. | 443 | |**nginxAgent.tlsSkipVerify** | Skip TLS verification for the NGINX One Console endpoint. Requires NGINX Agent 3.x. | false | |**nginxAgent.instanceGroup** | Set a custom Instance Group name for the deployment, shown when connected to NGINX Instance Manager. `nginx-ingress.controller.fullname` will be used if not set. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.host** | FQDN or IP for connecting to NGINX Ingress Controller. Required when `nginxAgent.enable` is set to `true`. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.grpcPort** | Port for connecting to NGINX Ingress Controller. Requires NGINX Agent 2.x. | 443 | |**nginxAgent.instanceManager.sni** | Server Name Indication for Instance Manager. See the NGINX Agent [docs](/agent/configuration/encrypt-communication.md) for more details. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.tls.enable** | Enable TLS for Instance Manager connection. Requires NGINX Agent 2.x. | true | |**nginxAgent.instanceManager.tls.skipVerify** | Skip certification verification for Instance Manager connection. Requires NGINX Agent 2.x. | false | |**nginxAgent.instanceManager.tls.caSecret** | Name of `nginx.org/ca` secret used for verification of Instance Manager TLS. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.tls.secret** | Name of `kubernetes.io/tls` secret with a TLS certificate and key for using mTLS between NGINX Agent and Instance Manager. See the NGINX Instance Manager [docs](/nim/system-configuration/secure-traffic.md#mutual-client-certificate-authentication-setup-mtls) and the NGINX Agent [docs](/agent/configuration/encrypt-communication.md) for more details. Requires NGINX Agent 2.x. | "" | |**nginxAgent.syslog.host** | Address for NGINX Agent to run syslog listener. Requires NGINX Agent 2.x. | 127.0.0.1 | |**nginxAgent.syslog.port** | Port for NGINX Agent to run syslog listener. Requires NGINX Agent 2.x. | 1514 | |**nginxAgent.napMonitoring.collectorBufferSize** | Buffer size for collector. Will contain log lines and parsed log lines. Requires NGINX Agent 2.x. | 50000 | |**nginxAgent.napMonitoring.processorBufferSize** | Buffer size for processor. Will contain log lines and parsed log lines. Requires NGINX Agent 2.x. | 50000 | |**nginxAgent.customConfigMap** | The name of a custom ConfigMap to use instead of the one provided by default. Requires NGINX Agent 2.x.| "" | | **prometheus.create** | Expose NGINX or NGINX Plus metrics in the Prometheus format. | true | | **prometheus.port** | Configures the port to scrape the metrics. | 9113 | | **prometheus.scheme** | Configures the HTTP scheme to use for connections to the Prometheus endpoint. | http | | **prometheus.secret** | The namespace / name of a Kubernetes TLS Secret. If specified, this secret is used to secure the Prometheus endpoint with TLS connections. | "" | | **prometheus.service.create** | Create a Headless service to expose prometheus metrics. Requires `prometheus.create`. | false | | **prometheus.service.labels** | Kubernetes object labels to attach to the service object. | {service: "nginx-ingress-prometheus-service"} | | **prometheus.serviceMonitor.create** | Create a ServiceMonitor custom resource. Requires ServiceMonitor CRD to be installed. For the latest CRD, check the latest release on the [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator) GitHub repo under `example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml` | false | | **prometheus.serviceMonitor.labels** | Kubernetes object labels to attach to the serviceMonitor object. | {} | | **prometheus.serviceMonitor.selectorMatchLabels** | A set of labels to allow the selection of endpoints for the ServiceMonitor. | {service: "nginx-ingress-prometheus-service"} | | **prometheus.serviceMonitor.endpoints** | A list of endpoints allowed as part of this ServiceMonitor. | [port: prometheus] | | **rbac.create** | Configures RBAC. | true | | **rbac.clusterrole.create** | Configures creation of ClusterRole. Creation can be disabled when more fine-grained control over RBAC is required. For example when controller.watchNamespace is used. | true | | **serviceInsight.create** | Expose NGINX Plus Service Insight endpoint. | false | | **serviceInsight.port** | Configures the port to expose endpoints. | 9114 | | **serviceInsight.scheme** | Configures the HTTP scheme to use for connections to the Service Insight endpoint. | http | | **serviceInsight.secret** | The namespace / name of a Kubernetes TLS Secret. If specified, this secret is used to secure the Service Insight endpoint with TLS connections. | "" | | **serviceNameOverride** | Used to prevent cloud load balancers from being replaced due to service name change during helm upgrades. | "" | -------------------------------------------------------------------------------- title: "NGINX Ingress Controller LTS Helm chart parameters" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/helm/parameters/index.md -------------------------------------------------------------------------------- This page describes the Helm chart parameters of F5 NGINX Ingress Controller LTSs. It is a reference for each parameter name, a description of the parameter, and any default values. ## NGINX Ingress Controller LTS Helm chart parameters **note:** The [values.yaml](https://github.com/nginx/kubernetes-ingress/blob/main/charts/nginx-ingress/values.yaml) file of the Helm chart is a single source of truth and includes parameter descriptions: the table may become outdated. The [values.schema.json](https://github.com/nginx/kubernetes-ingress/blob/main/charts/nginx-ingress/values.schema.json) file can be used to inspect how values are formatted. | Parameter | Description | Default | | ----------| ----------- | ------- | | **controller.name** | The name of the NGINX Ingress Controller LTS deployment, daemonset, or statefulset. | Autogenerated | | **controller.kind** | The kind of the NGINX Ingress Controller LTS installation - deployment, daemonset, or statefulset. | deployment | | **controller.annotations** | Allows for setting of `annotations` for deployment, daemonset, or statefulset. | {} | | **controller.nginxplus** | Deploys NGINX Ingress Controller LTS for NGINX Plus. | true | | **controller.mgmt.licenseTokenSecretName** | Configures the secret used in the [license_token](https://nginx.org/en/docs/ngx_mgmt_module.html#license_token) directive. This key assumes the secret is in the Namespace that NGINX Ingress Controller LTS is deployed in. The secret must be of type `nginx.com/license` with the base64 encoded JWT in the `license.jwt` key. | license-token | | **controller.mgmt.enforceInitialReport** | Configures the [enforce_initial_report](https://nginx.org/en/docs/ngx_mgmt_module.html#enforce_initial_report) directive, which enables or disables the 180-day grace period for sending the initial usage report. | false | | **controller.mgmt.usageReport.endpoint** | Configures the endpoint of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This is used to configure the endpoint NGINX uses to send usage reports to NGINX Instance Manager . | product.connect.nginx.com | | **controller.mgmt.usageReport.interval** | Configures the interval of the [usage_report](https://nginx.org/en/docs/ngx_mgmt_module.html#usage_report) directive. This specifies the frequency that usage reports are sent. Only seconds(s), minutes(m), and hours(h) are allowed and must be between 60s and 24h. | 1h | | **controller.mgmt.usageReport.proxyHost** | Configures the host name of the [proxy](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy) directive with optional port. | N/A | | **controller.mgmt.usageReport.proxyCredentialsSecretName** | Configures the [proxy_username](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy_username) directive as well as the [proxy_password](https://nginx.org/en/docs/ngx_mgmt_module.html#proxy_password) directive using a Kubernetes Opaque Secret. The Secret must contain `username` and `password` fields. | N/A | | **controller.mgmt.sslVerify** | Configures the [ssl_verify](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_verify) directive, which enables or disables verification of the usage reporting endpoint certificate. | true | | **controller.mgmt.resolver.ipv6** | Configures whether the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will look up IPv6 addresses. | true | | **controller.mgmt.resolver.valid** | Configures an [NGINX time](https://nginx.org/en/docs/syntax.html) that the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive will override the TTL value of responses from nameservers with. | N/A | | **controller.mgmt.resolver.addresses** | Configures addresses used in the mgmt block [resolver](https://nginx.org/en/docs/ngx_mgmt_module.html#resolver) directive. This field takes a list of addresses. | N/A | | **controller.mgmt.sslCertificateSecretName** | Configures the secret used to create the `ssl_certificate` and `ssl_certificate_key` directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller LTS is deployed in. The secret must be of type `kubernetes.io/tls` | N/A | | **controller.mgmt.sslTrustedCertificateSecretName** | Configures the secret used to create the file(s) referenced the in [ssl_trusted_certifcate](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_trusted_certificate), and [ssl_crl](https://nginx.org/en/docs/ngx_mgmt_module.html#ssl_crl) directives. This key assumes the secret is in the Namespace that NGINX Ingress Controller LTS is deployed in. The secret must be of type `nginx.org/ca`, where the `ca.crt` key contains a base64 encoded trusted cert, and the optional `ca.crl` key can contain a base64 encoded CRL. If the optional `ca.crl` key is supplied, it will configure the NGINX `ssl_crl` directive. | N/A | | **controller.mgmt.configMapName** | Allows changing the name of the MGMT config map. The name should not include a namespace| Autogenerated | | **controller.nginxReloadTimeout** | The timeout in milliseconds which the NGINX Ingress Controller LTS will wait for a successful NGINX reload after a change or at the initial start. | 60000 | | **controller.hostNetwork** | Enables NGINX Ingress Controller LTS pods to use the host's network namespace. | false | | **controller.dnsPolicy** | DNS policy for NGINX Ingress Controller LTS pods. | ClusterFirst | | **controller.nginxDebug** | Enables debugging for NGINX. Uses the `nginx-debug` binary. Requires `error-log-level: debug` in the ConfigMap via `controller.config.entries`. | false | | **controller.logLevel** | The log level of NGINX Ingress Controller LTS. | info | | **controller.logFormat** | The log format of NGINX Ingress Controller LTS. | glog | | **controller.directiveAutoAdjust** | Automatically adjusts NGINX buffer directives to prevent configuration errors. | false | | **controller.image.digest** | The image digest of NGINX Ingress Controller LTS. | None | | **controller.image.repository** | The image repository of NGINX Ingress Controller LTS. | private-registry.nginx.com/nginx-ic/nginx-plus-ingress | | **controller.image.tag** | The tag of the NGINX Ingress Controller LTS image. | | | **controller.image.pullPolicy** | The pull policy for the NGINX Ingress Controller LTS image. | IfNotPresent | | **controller.lifecycle** | The lifecycle of the NGINX Ingress Controller LTS pods. | {} | | **controller.customConfigMap** | The name of the custom ConfigMap used by the NGINX Ingress Controller LTS. If set, then the default config is ignored. | "" | | **controller.config.name** | The name of the ConfigMap used by the NGINX Ingress Controller LTS. | Autogenerated | | **controller.config.annotations** | The annotations of the NGINX Ingress Controller LTS configmap. | {} | | **controller.config.entries** | The entries of the ConfigMap for customizing NGINX configuration. See [ConfigMap resource docs](/nic/lts/configuration/global-configuration/configmap-resource.md) for the list of supported ConfigMap keys. | {} | | **controller.customPorts** | A list of custom ports to expose on the NGINX Ingress Controller LTS pod. Follows the conventional Kubernetes yaml syntax for container ports. | [] | | **controller.defaultTLS.cert** | The base64-encoded TLS certificate for the default HTTPS server. **Note:** It is recommended that you specify your own certificate. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server. | | **controller.defaultTLS.key** | The base64-encoded TLS key for the default HTTPS server. **Note:** It is recommended that you specify your own key. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server. | | **controller.defaultTLS.secret** | The secret with a TLS certificate and key for the default HTTPS server. The value must follow the following format: `/`. Used as an alternative to specifying a certificate and key using `controller.defaultTLS.cert` and `controller.defaultTLS.key` parameters. **Note:** Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server. | None | | **controller.wildcardTLS.cert** | The base64-encoded TLS certificate for every Ingress/VirtualServer host that has TLS enabled but no secret specified. If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection. | None | | **controller.wildcardTLS.key** | The base64-encoded TLS key for every Ingress/VirtualServer host that has TLS enabled but no secret specified. If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection. | None | | **controller.wildcardTLS.secret** | The secret with a TLS certificate and key for every Ingress/VirtualServer host that has TLS enabled but no secret specified. The value must follow the following format: `/`. Used as an alternative to specifying a certificate and key using `controller.wildcardTLS.cert` and `controller.wildcardTLS.key` parameters. | None | | **controller.nodeSelector** | The node selector for pod assignment for NGINX Ingress Controller LTS pods. | {} | | **controller.terminationGracePeriodSeconds** | The termination grace period of the NGINX Ingress Controller LTS pod. | 30 | | **controller.tolerations** | The tolerations of NGINX Ingress Controller LTS pods. | [] | | **controller.affinity** | The affinity of NGINX Ingress Controller LTS pods. | {} | | **controller.topologySpreadConstraints** | The topology spread constraints of the NGINX Ingress Controller LTS pods. | {} | | **controller.env** | The additional environment variables to be set on the NGINX Ingress Controller LTS pods. | [] | | **controller.volumes** | The volumes of NGINX Ingress Controller LTS pods. | [] | | **controller.volumeMounts** | The volumeMounts of NGINX Ingress Controller LTS pods. | [] | | **controller.initContainers** | InitContainers for NGINX Ingress Controller LTS pods. | [] | | **controller.extraContainers** | Extra (eg. sidecar) containers for NGINX Ingress Controller LTS pods. | [] | | **controller.podSecurityContext**| The SecurityContext for Ingress Controller pods. | "seccompProfile": {"type": "RuntimeDefault"} | | **controller.securityContext** | The SecurityContext for Ingress Controller container. | {} | | **controller.initContainerSecurityContext** | The SecurityContext for NGINX Ingress Controller LTS init container when `readOnlyRootFilesystem` is enabled by either setting `controller.securityContext.readOnlyRootFilesystem` or `controller.readOnlyRootFilesystem`to `true`. | {} | | **controller.resources** | The resources of NGINX Ingress Controller LTS pods. | requests: cpu=100m,memory=128Mi | | **controller.initContainerResources** | The resources of the init container which is used when `readOnlyRootFilesystem` is enabled by either setting `controller.securityContext.readOnlyRootFilesystem` or `controller.readOnlyRootFilesystem`to `true`. | requests: cpu=100m,memory=128Mi | | **controller.replicaCount** | The number of replicas of the NGINX Ingress Controller LTS deployment. | 1 | | **controller.ingressClass.name** | A class of NGINX Ingress Controller LTS. An IngressClass resource with the name equal to the class must be deployed. Otherwise, the NGINX Ingress Controller LTS will fail to start. the NGINX Ingress Controller LTS only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class. the NGINX Ingress Controller LTS processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field for all versions of Kubernetes. | nginx | | **controller.ingressClass.create** | Creates a new IngressClass object with the name `controller.ingressClass.name`. Set to `false` to use an existing ingressClass created using `kubectl` with the same name. If you use `helm upgrade`, do not change the values from the previous release as helm will delete IngressClass objects managed by helm. If you are upgrading from a release earlier than , do not set the value to false. | true | | **controller.ingressClass.setAsDefaultIngress** | New Ingresses without an `"ingressClassName"` field specified will be assigned the class specified in `controller.ingressClass.name`. Requires `controller.ingressClass.create`. | false | | **controller.watchNamespace** | Comma separated list of namespaces the NGINX Ingress Controller LTS should watch for resources. By default the NGINX Ingress Controller LTS watches all namespaces. Mutually exclusive with `controller.watchNamespaceLabel`. Please note that if configuring multiple namespaces using the Helm cli `--set` option, the string needs to wrapped in double quotes and the commas escaped using a backslash - e.g. `--set controller.watchNamespace="default\,nginx-ingress"`. | "" | | **controller.watchNamespaceLabel** | Configures NGINX Ingress Controller LTS to watch only those namespaces with label foo=bar. By default the NGINX Ingress Controller LTS watches all namespaces. Mutually exclusive with `controller.watchNamespace`. | "" | | **controller.watchSecretNamespace** | Comma separated list of namespaces the NGINX Ingress Controller LTS should watch for resources of type Secret. If this arg is not configured, the NGINX Ingress Controller LTS watches the same namespaces for all resources, see `controller.watchNamespace` and `controller.watchNamespaceLabel`. All namespaces included with this argument must be part of either `controller.watchNamespace` or `controller.watchNamespaceLabel`. Please note that if configuring multiple namespaces using the Helm cli `--set` option, the string needs to wrapped in double quotes and the commas escaped using a backslash - e.g. `--set controller.watchSecretNamespace="default\,nginx-ingress"`. | "" | | **controller.enableCustomResources** | Enable the custom resources. | true | | **controller.enableOIDC** | Enable OIDC policies. | false | | **controller.enableTLSPassthrough** | Enable TLS Passthrough on default port 443. Requires `controller.enableCustomResources`. | false | | **controller.tlsPassThroughPort** | Set the port for the TLS Passthrough. Requires `controller.enableCustomResources` and `controller.enableTLSPassthrough`. | 443 | | **controller.enableCertManager** | Enable x509 automated certificate management for VirtualServer resources using cert-manager (cert-manager.io). Requires `controller.enableCustomResources`. | false | | **controller.enableExternalDNS** | Enable integration with ExternalDNS for configuring public DNS entries for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Requires `controller.enableCustomResources`. | false | | **controller.globalConfiguration.create** | Creates the GlobalConfiguration custom resource. Requires `controller.enableCustomResources`. | false | | **controller.globalConfiguration.spec** | The spec of the GlobalConfiguration for defining the global configuration parameters of the NGINX Ingress Controller LTS. | {} | | **controller.enableSnippets** | Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources. | false | | **controller.healthStatus** | Add a location "/nginx-health" to the default server. The location responds with the 200 status code for any request. Useful for external health-checking of the NGINX Ingress Controller LTS. | false | | **controller.healthStatusURI** | Sets the URI of health status location in the default server. Requires `controller.healthStatus`. | "/nginx-health" | | **controller.nginxStatus.enable** | Enable the NGINX Plus status API. | true | | **controller.nginxStatus.port** | Set the port where the NGINX Plus API is exposed. | 8080 | | **controller.nginxStatus.allowCidrs** | Add IP/CIDR blocks to the allow list for the NGINX Plus API. Separate multiple IP/CIDR by commas. | 127.0.0.1,::1 | | **controller.priorityClassName** | The PriorityClass of NGINX Ingress Controller LTS pods. | None | | **controller.service.create** | Creates a service to expose NGINX Ingress Controller LTS pods. | true | | **controller.service.type** | The type of service to create for NGINX Ingress Controller LTS. | LoadBalancer | | **controller.service.externalTrafficPolicy** | The externalTrafficPolicy of the service. The value Local preserves the client source IP. | Local | | **controller.service.annotations** | The annotations of the NGINX Ingress Controller LTS service. | {} | | **controller.service.extraLabels** | The extra labels of the service. | {} | | **controller.service.loadBalancerIP** | The static IP address for the load balancer. Requires `controller.service.type` set to `LoadBalancer`. The cloud provider must support this feature. | "" | | **controller.service.loadBalancerClass** | This property allows the user to define a specific load balancer implementation or configuration class for the service. Requires `controller.service.type` set to `LoadBalancer`. The cloud provider must support this feature. | "" | | **controller.service.externalIPs** | The list of external IPs for the NGINX Ingress Controller LTS service. | [] | | **controller.service.clusterIP** | The clusterIP for the NGINX Ingress Controller LTS service, autoassigned if not specified. | "" | | **controller.service.loadBalancerSourceRanges** | The IP ranges (CIDR) that are allowed to access the load balancer. Requires `controller.service.type` set to `LoadBalancer`. The cloud provider must support this feature. | [] | | **controller.service.name** | The name of the service. | Autogenerated | | **controller.service.customPorts** | A list of custom ports to expose through the NGINX Ingress Controller LTS service. Follows the conventional Kubernetes yaml syntax for service ports. | [] | | **controller.service.httpPort.enable** | Enables the HTTP port for the NGINX Ingress Controller LTS service. | true | | **controller.service.httpPort.port** | The HTTP port of the NGINX Ingress Controller LTS service. | 80 | | **controller.service.httpPort.nodePort** | The custom NodePort for the HTTP port. Requires `controller.service.type` set to `NodePort`. | "" | | **controller.service.httpPort.targetPort** | The target port of the HTTP port of the NGINX Ingress Controller LTS service. | 80 | | **controller.service.httpsPort.enable** | Enables the HTTPS port for the NGINX Ingress Controller LTS service. | true | | **controller.service.httpsPort.port** | The HTTPS port of the NGINX Ingress Controller LTS service. | 443 | | **controller.service.httpsPort.nodePort** | The custom NodePort for the HTTPS port. Requires `controller.service.type` set to `NodePort`. | "" | | **controller.service.httpsPort.targetPort** | The target port of the HTTPS port of the NGINX Ingress Controller LTS service. | 443 | | **controller.service.sessionAffinity.enable** | Enable session affinity. | `false` | | **controller.service.sessionAffinity.type** | Session affinity type. Valid values: `None`, `ClientIP`. | `ClientIP` | | **controller.service.sessionAffinity.timeout** | Session affinity timeout in seconds. | 3600 | | **controller.serviceAccount.annotations** | The annotations of the NGINX Ingress Controller LTS service account. | {} | | **controller.serviceAccount.name** | The name of the service account of the NGINX Ingress Controller LTS pods. Used for RBAC. | Autogenerated | | **controller.serviceAccount.imagePullSecretName** | The name of the secret containing docker registry credentials. Secret must exist in the same namespace as the helm release. | "" | | **controller.serviceAccount.imagePullSecretsNames** | The list of secret names containing docker registry credentials. Secret must exist in the same namespace as the helm release. | [] | | **controller.reportIngressStatus.enable** | Updates the address field in the status of Ingress resources with an external address of the NGINX Ingress Controller LTS. You must also specify the source of the external address either through an external service via `controller.reportIngressStatus.externalService`, `controller.reportIngressStatus.ingressLink` or the `external-status-address` entry in the ConfigMap via `controller.config.entries`. **Note:** `controller.config.entries.external-status-address` takes precedence over the others. | true | | **controller.reportIngressStatus.externalService** | Specifies the name of the service with the type LoadBalancer through which the NGINX Ingress Controller LTS is exposed externally. The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. `controller.reportIngressStatus.enable` must be set to `true`. The default is autogenerated and enabled when `controller.service.create` is set to `true` and `controller.service.type` is set to `LoadBalancer`. | Autogenerated | | **controller.reportIngressStatus.ingressLink** | Specifies the name of the IngressLink resource, which exposes the NGINX Ingress Controller LTS pods via a BIG-IP system. The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources. `controller.reportIngressStatus.enable` must be set to `true`. | "" | | **controller.reportIngressStatus.enableLeaderElection** | Enable Leader election to avoid multiple replicas of the controller reporting the status of Ingress resources. `controller.reportIngressStatus.enable` must be set to `true`. | true | | **controller.reportIngressStatus.leaderElectionLockName** | Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. controller.reportIngressStatus.enableLeaderElection must be set to true. | Autogenerated | | **controller.reportIngressStatus.annotations** | The annotations of the leader election configmap. | {} | | **controller.pod.annotations** | The annotations of the NGINX Ingress Controller LTS pod. | {} | | **controller.pod.extraLabels** | The additional extra labels of the NGINX Ingress Controller LTS pod. | {} | | **controller.readyStatus.enable** | Enables the readiness endpoint `"/nginx-ready"`. The endpoint returns a success code when NGINX has loaded all the config after the startup. This also configures a readiness probe for the NGINX Ingress Controller LTS pods that uses the readiness endpoint. | true | | **controller.readyStatus.port** | The HTTP port for the readiness endpoint. | 8081 | | **controller.readyStatus.initialDelaySeconds** | The number of seconds after the NGINX Ingress Controller LTS pod has started before readiness probes are initiated. | 0 | | **controller.startupStatus.enable** | Enables the startup probe for the Ingress Controller. | false | | **controller.startupStatus.port** | The port where the startup endpoint is exposed. This is a required field if `controller.startupStatus.enable` is set to true. | N/A | | **controller.startupStatus.path** | The path to the startup endpoint. This is a required field if `controller.startupStatus.enable` is set to true. | N/A | | **controller.startupStatus.initialDelaySeconds** | The number of seconds after the Ingress Controller pod has started before startup probes are initiated. | N/A | | **controller.startupStatus.periodSeconds** | The number of seconds between each startup probe. | N/A | | **controller.startupStatus.timeoutSeconds** | The number of seconds after which the startup probe times out. | N/A | | **controller.startupStatus.successThreshold** | Minimum consecutive successes for the probe to be considered successful. | N/A | | **controller.startupStatus.failureThreshold** | When a probe fails, Kubernetes will try failureThreshold times before giving up. | N/A | | **controller.enableLatencyMetrics** | Enable collection of latency metrics for upstreams. Requires `prometheus.create`. | false | | **controller.minReadySeconds** | Specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. [docs](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#min-ready-seconds) | 0 | | **controller.autoscaling.enabled** | Enables HorizontalPodAutoscaling. | false | | **controller.autoscaling.create** | Create the HorizontalPodAutoscaler resource. This can be set to false to manage the HPA externally. | true | | **controller.autoscaling.annotations** | The annotations of the NGINX Ingress Controller LTS HorizontalPodAutoscaler. | {} | | **controller.autoscaling.behavior** | Behavior configuration for the HPA. | {} | | **controller.autoscaling.minReplicas** | Minimum number of replicas for the HPA. | 1 | | **controller.autoscaling.maxReplicas** | Maximum number of replicas for the HPA. | 3 | | **controller.autoscaling.targetCPUUtilizationPercentage** | The target CPU utilization percentage. | 50 | | **controller.autoscaling.targetMemoryUtilizationPercentage** | The target memory utilization percentage. | 50 | | **controller.podDisruptionBudget.enabled** | Enables PodDisruptionBudget. | false | | **controller.podDisruptionBudget.annotations** | The annotations of the NGINX Ingress Controller LTS pod disruption budget | {} | | **controller.podDisruptionBudget.minAvailable** | The number of Ingress Controller pods that should be available. This is a mutually exclusive setting with "maxUnavailable". | 0 | | **controller.podDisruptionBudget.maxUnavailable** | The number of Ingress Controller pods that can be unavailable. This is a mutually exclusive setting with "minAvailable". | 0 | | **controller.strategy** | Specifies the strategy used to replace old Pods with new ones. Docs for [Deployment update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy), [Daemonset update strategy](https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/#daemonset-update-strategy) and [StatefulSet update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies) | {} | | **controller.statefulset.podManagementPolicy** | Controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. Available options: `OrderedReady` or `Parallel`. Only applies when `controller.kind` is set to `statefulset`. | `OrderedReady` | | **controller.statefulset.persistentVolumeClaimRetentionPolicy.whenDeleted** | Controls the retention policy for PVCs when the StatefulSet is deleted. Available options: `Retain` or `Delete`. Only applies when `controller.kind` is set to `statefulset`. | `Retain` | | **controller.statefulset.persistentVolumeClaimRetentionPolicy.whenScaled** | Controls the retention policy for PVCs when the StatefulSet is scaled down. Available options: `Retain` or `Delete`. Only applies when `controller.kind` is set to `statefulset`. | `Retain` | | **controller.statefulset.nginxCachePVC.size** | The size of the persistent volume claim for NGINX cache storage. Only applies when `controller.kind` is set to `statefulset`. | `256Mi` | | **controller.statefulset.nginxCachePVC.storageClass** | The storage class for the persistent volume claim. Only applies when `controller.kind` is set to `statefulset`. | `""` | | **controller.statefulset.nginxCachePVC.accessModes** | The access modes for the persistent volume claim. Only applies when `controller.kind` is set to `statefulset`. | `["ReadWriteOnce"]` | | **controller.disableIPV6** | Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack. | false | | **controller.defaultHTTPListenerPort** | Sets the port for the HTTP `default_server` listener. | 80 | | **controller.defaultHTTPSListenerPort** | Sets the port for the HTTPS `default_server` listener. | 443 | | **controller.readOnlyRootFilesystem** | Configure root filesystem as read-only and add volumes for temporary data. Three major releases after 3.5.x this argument will be moved permanently to the `controller.securityContext` section. | false | | **controller.enableSSLDynamicReload** | Enable lazy loading for SSL Certificates. | true | | **controller.telemetryReporting.enable** | Enable telemetry reporting. | true | | **controller.enableWeightChangesDynamicReload** | Enable weight changes without reloading the NGINX configuration. May require increasing `map_hash_bucket_size`, `map_hash_max_size`, `variable_hash_bucket_size`, and `variable_hash_max_size` in the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource.md) if there are many two-way splits. Requires `controller.nginxplus` | false | |**nginxAgent.enable** | Enable NGINX Agent 3.x to allow [connecting to NGINX One Console](/nginx-one-console/k8s/add-nic.md) . | false | |**nginxAgent.logLevel** | Log level for NGINX Agent. | "error" | |**nginxAgent.dataplaneKeySecretName** | Name of the Kubernetes Secret containing the Data Plane key used to authenticate to NGINX One Console. Learn more [here](/nginx-one-console/k8s/add-nic.md). Required when `nginxAgent.enable` is set to `true`. Requires NGINX Agent 3.x. | "" | |**nginxAgent.endpointHost** | Domain or IP address for the NGINX One Console. Requires NGINX Agent 3.x. | "agent.connect.nginx.com" | |**nginxAgent.endpointPort** | Port for the NGINX One Console endpoint. Requires NGINX Agent 3.x. | 443 | |**nginxAgent.tlsSkipVerify** | Skip TLS verification for the NGINX One Console endpoint. Requires NGINX Agent 3.x. | false | |**nginxAgent.instanceGroup** | Set a custom Instance Group name for the deployment, shown when connected to NGINX Instance Manager. `nginx-ingress.controller.fullname` will be used if not set. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.host** | FQDN or IP for connecting to NGINX Ingress Controller LTS. Required when `nginxAgent.enable` is set to `true`. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.grpcPort** | Port for connecting to NGINX Ingress Controller LTS. Requires NGINX Agent 2.x. | 443 | |**nginxAgent.instanceManager.sni** | Server Name Indication for Instance Manager. See the NGINX Agent [docs](/agent/configuration/encrypt-communication.md) for more details. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.tls.enable** | Enable TLS for Instance Manager connection. Requires NGINX Agent 2.x. | true | |**nginxAgent.instanceManager.tls.skipVerify** | Skip certification verification for Instance Manager connection. Requires NGINX Agent 2.x. | false | |**nginxAgent.instanceManager.tls.caSecret** | Name of `nginx.org/ca` secret used for verification of Instance Manager TLS. Requires NGINX Agent 2.x. | "" | |**nginxAgent.instanceManager.tls.secret** | Name of `kubernetes.io/tls` secret with a TLS certificate and key for using mTLS between NGINX Agent and Instance Manager. See the NGINX Instance Manager [docs](/nim/system-configuration/secure-traffic.md#mutual-client-certificate-authentication-setup-mtls) and the NGINX Agent [docs](/agent/configuration/encrypt-communication.md) for more details. Requires NGINX Agent 2.x. | "" | |**nginxAgent.syslog.host** | Address for NGINX Agent to run syslog listener. Requires NGINX Agent 2.x. | 127.0.0.1 | |**nginxAgent.syslog.port** | Port for NGINX Agent to run syslog listener. Requires NGINX Agent 2.x. | 1514 | |**nginxAgent.napMonitoring.collectorBufferSize** | Buffer size for collector. Will contain log lines and parsed log lines. Requires NGINX Agent 2.x. | 50000 | |**nginxAgent.napMonitoring.processorBufferSize** | Buffer size for processor. Will contain log lines and parsed log lines. Requires NGINX Agent 2.x. | 50000 | |**nginxAgent.customConfigMap** | The name of a custom ConfigMap to use instead of the one provided by default. Requires NGINX Agent 2.x.| "" | | **nginxServiceMesh.enable** | Enable integration with NGINX Service Mesh. See the NGINX Service Mesh docs for more details. Requires `controller.nginxplus`. | false | | **nginxServiceMesh.enableEgress** | Enable NGINX Service Mesh workloads to route egress traffic through the NGINX Ingress Controller LTS. See the NGINX Service Mesh docs for more details. Requires `nginxServiceMesh.enable`. | false | | **prometheus.create** | Expose NGINX Plus metrics in the Prometheus format. | true | | **prometheus.port** | Configures the port to scrape the metrics. | 9113 | | **prometheus.scheme** | Configures the HTTP scheme to use for connections to the Prometheus endpoint. | http | | **prometheus.secret** | The namespace / name of a Kubernetes TLS Secret. If specified, this secret is used to secure the Prometheus endpoint with TLS connections. | "" | | **prometheus.service.create** | Create a Headless service to expose prometheus metrics. Requires `prometheus.create`. | false | | **prometheus.service.labels** | Kubernetes object labels to attach to the service object. | {service: "nginx-ingress-prometheus-service"} | | **prometheus.serviceMonitor.create** | Create a ServiceMonitor custom resource. Requires ServiceMonitor CRD to be installed. For the latest CRD, check the latest release on the [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator) GitHub repo under `example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml` | false | | **prometheus.serviceMonitor.labels** | Kubernetes object labels to attach to the serviceMonitor object. | {} | | **prometheus.serviceMonitor.selectorMatchLabels** | A set of labels to allow the selection of endpoints for the ServiceMonitor. | {service: "nginx-ingress-prometheus-service"} | | **prometheus.serviceMonitor.endpoints** | A list of endpoints allowed as part of this ServiceMonitor. | [port: prometheus] | | **rbac.create** | Configures RBAC. | true | | **rbac.clusterrole.create** | Configures creation of ClusterRole. Creation can be disabled when more fine-grained control over RBAC is required. For example when controller.watchNamespace is used. | true | | **serviceInsight.create** | Expose NGINX Plus Service Insight endpoint. | false | | **serviceInsight.port** | Configures the port to expose endpoints. | 9114 | | **serviceInsight.scheme** | Configures the HTTP scheme to use for connections to the Service Insight endpoint. | http | | **serviceInsight.secret** | The namespace / name of a Kubernetes TLS Secret. If specified, this secret is used to secure the Service Insight endpoint with TLS connections. | "" | | **serviceNameOverride** | Used to prevent cloud load balancers from being replaced due to service name change during helm upgrades. | "" | -------------------------------------------------------------------------------- title: "Ingress-specific policy behavior" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/ingress-specific-behavior/index.md -------------------------------------------------------------------------------- Ingress resources support Policy resources, but with some differences from VirtualServer and VirtualServerRoute resources. Review the following behavior before you attach a policy to an Ingress. ## Ingress doesn't support every policy type Ingress policy support is narrower than VirtualServer support. If you need route-level control for features like JWT, OIDC (NJS), cache, or rate limiting, use VirtualServer and VirtualServerRoute instead. ## WAF on Ingress must use "nginx.com/policies" WAF is a Plus-only feature. When you use a WAF policy with Ingress, reference it through: ```yaml metadata: annotations: nginx.com/policies: waf-policy ``` Don't attach WAF to Ingress with `nginx.org/policies`. **important:** On Ingress, `nginx.org/policies` and `nginx.com/policies` aren't interchangeable. Reference WAF policies only through `nginx.com/policies`. ## Egress mTLS on Ingress only configures TLS parameters `egressMTLS` defines how NGINX authenticates to the upstream and verifies the upstream certificate. It doesn't switch the upstream transport from plain HTTP to HTTPS. For Ingress, you still need the upstream connection itself to use TLS. Common ways to do this include: - `nginx.org/ssl-services` for HTTPS upstreams - `nginx.org/grpc-services` for gRPC upstreams For example: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webapp-ingress annotations: nginx.org/policies: egress-mtls-policy nginx.org/ssl-services: "secure-app" spec: ingressClassName: nginx rules: - host: webapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: secure-app port: number: 8443 ``` Without upstream TLS, the egress mTLS policy has TLS settings but no TLS connection to apply them to. NGINX then typically sends plain HTTP to an HTTPS upstream port. **Note:** For Ingress, `egressMTLS` configures how NGINX uses TLS when connecting to the upstream. It doesn't decide whether the upstream connection uses TLS. Configure that separately, for example with `nginx.org/ssl-services` or `nginx.org/grpc-services`. ## Mergeable Ingress behavior For mergeable Ingress: - Policies on the master apply to inherited minion configuration. - Policies on the minion override policies of the same type from the master. This matches the general expectation that a more specific resource overrides a broader one. For `egressMTLS`, there is one extra detail: - The master or standard Ingress policy applies at server scope. - The minion override applies at location scope, so the minion can replace the value set by the master. ## What's next Learn about [policy precedence and override rules](/nic/configuration/policy-resource/precedence-and-overrides.md). -------------------------------------------------------------------------------- title: "Troubleshoot F5 DoS for NGINX" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-dos/troubleshoot-app-protect-dos/index.md -------------------------------------------------------------------------------- This document describes how to troubleshoot problems when using NGINX Ingress Controller and the App Protect DoS module. To troubleshoot other parts of NGINX Ingress Controller, check the [troubleshooting](/nic/troubleshooting/troubleshoot-common.md) section of the documentation. ## Potential problems The table below outlines potential problems with NGINX Ingress Controller when the App Protect DoS module is enabled. It suggests how to troubleshoot those problems with methods explained in the next section. |Problem area | Symptom | Troubleshooting method | Common cause | | ---| ---| ---| --- | |Start | NGINX Ingress Controller fails to start. | Check the NGINX Ingress Controller logs. | Misconfigured DosProtectedResource, APDosLogConf or APDosPolicy. | |DosProtectedResource, APDosLogConf, APDosPolicy or Ingress Resource. | The configuration is not applied. | Check the events of the DosProtectedResource, APDosLogConf, APDosPolicy and Ingress Resource, check the Ingress Controller logs. | DosProtectedResource, APDosLogConf or APDosPolicy is invalid. | ## Troubleshooting ethods ### Checking NGINX Ingress Controller and App Protect DoS logs App Protect DoS logs are part of the NGINX Ingress Controller logs when the module is enabled. To check the Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs](/nic/troubleshooting/troubleshoot-common#checking-nginx-ingress-controller-logss) of the Troubleshooting guide. For App Protect DoS specific logs, look for messages starting with `APP_PROTECT_DOS`, such as: ```shell 2021/06/14 08:17:50 [notice] 242#242: APP_PROTECT_DOS { "event": "shared_memory_connected", "worker_pid": 242, "mode": "operational", "mode_changed": true } ``` ### Checking Ingress Resource Events Follow the steps of [Troubleshooting Ingress Resources](/nic/troubleshooting/troubleshoot-ingress). ### Checking VirtualServer Resource Events Follow the steps of [Troubleshooting VirtualServer Resources](/nic/troubleshooting/troubleshoot-virtualserver). ### Checking for DoSProtectedResource Events After you create or update an DosProtectedResource, you can immediately check if the NGINX configuration was successfully applied by NGINX: ```shell kubectl describe dosprotectedresource dos-protected Name: dos-protected Namespace: default Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 2s nginx-ingress-controller Configuration for default/dos-protected was added or updated ``` Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. If the DosProtectedResource refers to a missing resource, you should see a message like the following: ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 8s nginx-ingress-controller dos protected refers (default/dospolicy) to an invalid DosPolicy: DosPolicy default/dospolicy not found ``` This can be fixed by adding the missing resource. ### Checking for APDosLogConf Events After you create or update an APDosLogConf, you can immediately check if the NGINX configuration was successfully applied by NGINX: ```shell kubectl describe apdoslogconf logconf Name: logconf Namespace: default Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 11s nginx-ingress-controller AppProtectDosLogConfig default/logconf was added or updated ``` Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. ### Check events of APDosPolicy After you create or update an APDosPolicy, you can immediately check if the NGINX configuration was successfully applied by NGINX: ```shell kubectl describe apdospolicy dospolicy Name: dospolicy Namespace: default . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 2m25s nginx-ingress-controller AppProtectDosPolicy default/dospolicy was added or updated ``` The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. ## Run App Protect DoS in Debug log Mode When you configure NGINX Ingress Controller to use debug log mode, the setting also applies to the App Protect DoS module. See [Enable debugging for NGINX Ingress Controller ](/nic/troubleshooting/troubleshoot-common.md#enable-debugging-for-nginx-ingress-controller) for instructions. You can enable debug log mode to App Protect DoS module only by setting the `app-protect-dos-debug` [configmap](/nic/configuration/global-configuration/configmap-resource.md#modules). -------------------------------------------------------------------------------- title: "Troubleshoot F5 WAF for NGINX" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-waf-v5/troubleshoot-app-protect-waf/index.md -------------------------------------------------------------------------------- This document describes how to troubleshoot problems when using NGINX Ingress Controller and the F5 WAF for NGINX module version 5. For general troubleshooting of NGINX Ingress Controller, check the general [troubleshooting](/nic/troubleshooting/troubleshoot-common) documentation. ## Potential problems The table below categorizes some potential problems with NGINX Ingress Controller when F5 WAF for NGINX module is enabled. It suggests how to troubleshoot those problems, using one or more methods from the next section. |Problem area | Symptom | Troubleshooting method | Common cause | | ---| ---| ---| --- | |Start. | The Ingress Controller fails to start. | Check the logs. | Misconfigured policy bundle. | |Start | The configuration is not applied. | Check if a policy bundle is compiled using version of the compiler running in NGINX Ingress Controller. | Policy bundle is invalid. | |Start | The configuration is not applied. | Check if bundle is present in a volume. | Policy bundle is not present in the mounted volume. | |APLogConf, Policy or Ingress Resource. | The configuration is not applied. | Check the events of the APLogConf, Policy and Ingress Resource, check the logs, replace the policy bundle. | Policy bundle is invalid. | |[Bundle source](/nic/integrations/app-protect-waf-v5/bundle-sources.md). | Bundle not fetched, Warning event on Policy. | Check Policy events and status. Verify the bundle source URL and credentials are correct. | Invalid URL, authentication failure, or bundle not yet compiled on the management plane. | |[Bundle source](/nic/integrations/app-protect-waf-v5/bundle-sources.md). | VirtualServer or Ingress returns HTTP 500. | Check Policy status for bundle source errors. | Bundle source is unreachable or the bundle is not yet available. | |[Bundle source](/nic/integrations/app-protect-waf-v5/bundle-sources.md). | Policy not updating after bundle recompilation. | Verify ``enablePolling`` is ``true`` and ``pollInterval`` is at least ``1m``. | Polling is not enabled, or the poll interval is below the minimum. | ## Troubleshooting methods ### Check NGINX Ingress Controller and App Protect logs App Protect logs are part of NGINX Ingress Controller logs when the module is enabled. To check NGINX Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs](/nic/troubleshooting/troubleshoot-common#checking-the-ingress-controller-logs) of the Troubleshooting guide. For App Protect specific logs, look for messages starting with `APP_PROTECT`, for example: ```shell 2020/07/10 11:13:20 [notice] 17#17: APP_PROTECT { "event": "configuration_load_success", "software_version": "2.52.1", "completed_successfully":true,"attack_signatures_package":{"revision_datetime":"2020-06-18T10:11:32Z","version":"2020.06.18"}} ``` ### Check Ingress resource events Read the topic [Troubleshooting Ingress resources](/nic/troubleshooting/troubleshoot-ingress). ### Check APLogConf events After you create or update an APLogConf, you can immediately check if the NGINX configuration was successfully applied by NGINX: ```shell kubectl describe aplogconf logconf Name: logconf Namespace: default Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 11s nginx-ingress-controller AppProtectLogConfig default/logconf was added or updated ``` Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied. ### Check APPolicy events After you create or update an APPolicy, you can immediately check if the NGINX configuration was successfully applied by NGINX: ```shell kubectl describe appolicy dataguard-alarm Name: dataguard-alarm Namespace: default Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 2m25s nginx-ingress-controller AppProtectPolicy default/dataguard-alarm was added or updated ``` The events section has a *Normal* event with the *AddedOrUpdated reason*, indicating the policy was successfully accepted. ### Replace the policy **Note:** This method only applies if using [external references](/nginx-app-protect/v4/configuration/#external-references) If items on the external reference change but the spec of the APPolicy remains unchanged (even when re-applying the policy), Kubernetes will not detect the update. In this case you can force-replace the resource. This will remove the resource and add it again, triggering a reload. For example: ```shell kubectl replace appolicy -f your-policy-manifest.yaml --force ``` ### Check the availability of APPolicy external references **Note:** This method only applies if you're using [external references](/nginx-app-protect/v4/configuration/#external-references) in F5 WAF for NGINX policies. To check which servers host the external references of a policy: ```shell kubectl get appolicy mypolicy -o jsonpath='{.items[*].spec.policy.*.link}' | tr ' ' '\n' http://192.168.100.100/resources/headersettings.txt ``` You can check the total time a http request takes, in multiple ways eg. using curl: ```shell curl -w '%{time_total}' http://192.168.100.100/resources/headersettings.txt ``` ## Run App Protect in debug mode When you set NGINX Ingress Controller to use debug mode, the setting also applies to the F5 WAF for NGINX module. See [Running NGINX in the Debug Mode](/nic/troubleshooting/troubleshoot-common.md#enable-debugging-for-nginx-ingress-controller) for instructions. ## Known issues When using NGINX Ingress Controller with the F5 WAF for NGINX module, the following issues have been reported. The occurrence of these issues is commonly related to a higher number of Ingress Resources with App Protect being enabled in a cluster. ### Subrequests compatibility Policies that rely on NGINX subrequests (such as `ExternalAuth`, `APIKey`, `JWT` with remote JWKS fetching, `OIDC`, or `Cache` with `cacheBackgroundUpdate`) and F5 WAF for NGINX may not function as expected and may cause issues when applied together on the same route. When you make a change that requires NGINX to apply a new configuration, NGINX Ingress Controller reloads NGINX automatically. Without the F5 WAF for NGINX module enabled, usual reload times are around 150ms. If F5 WAF for NGINX module is enabled and is being used by any number of Ingress Resources, these reloads might take a few seconds instead. ### NGINX configuration drift If you are running more than one instance of NGINX Ingress Controller, the extended reload time may cause the NGINX configuration of your instances to be out of sync. This can occur because there is no order imposed on how NGINX Ingress Controller processes the Kubernetes Resources. The configurations will be the same after all instances have completed the reload. In order to reduce these inconsistencies, we advise that you do not apply changes to multiple resources handled by NGINX Ingress Controller at the same time. ### NGINX fails to start or reload The first time NGINX Ingress Controller starts, or whenever there is a change that requires reloading NGINX, NGINX Ingress Controller will verify if the reload was successful. The timeout for this verification is normally 4 seconds. When App Protect is enabled, this timeout increases to 20 seconds. This timeout should be more than enough to verify configurations. However, when numerous Ingress resources with App Protect enabled are handled by NGINX Ingress Controller at the same time, you may find that you need to extend the timeout further. Examples of when this might be necessary include: - You need to apply a large amount of Ingress Resources at once. - You are running NGINX Ingress Controller for the first time in a cluster where the Ingress resources with App Protect enabled are already present. You can increase this timeout by setting the `nginx-reload-timeout` [cli-argument](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-nginx-reload-timeout). When using the User Defined Signature feature, an update to an `APUserSig` requires more reload time from NGINX Plus compared with the other AppProtect resources. As a consequence, we recommend increasing the `nginx-reload-timeout` to 30 seconds if you're planning to use this feature. If you are using external references in your F5 WAF for NGINX policies, verify if the servers hosting the referenced resources are available and that their response time is as short as possible (see the Check the Availability of APPolicy External References section). If the references are not available during NGINX Ingress Controller startup, the pod will fail to start. In case the resources are not available during a reload, the reload will fail, and NGINX Plus will use the previous correct configuration. -------------------------------------------------------------------------------- title: "Use Helm to uninstall NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/install/helm/uninstall/index.md -------------------------------------------------------------------------------- This page describes how to use Helm to uninstall F5 NGINX Ingress Controller. It explains how to remove the chart, then remove the custom resource definitions (CRDs). By following these instructions, you will remove NGINX Ingress Controller from your Kubernetes cluster. ## Remove the Helm chart To uninstall NGINX Ingress Controller, you must first remove the chart. To remove a release named **\**, use the following command: ```shell helm uninstall ``` The command removes all the Kubernetes components associated with the release, then deletes the release itself. ## Remove the CRDs After removing the release, pull the chart sources: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress --untar --version ``` **Note:** The next command will delete all corresponding custom resources in your cluster across all namespaces. Before using it, check there are no custom resources that you want to keep, and that there are no other NGINX Ingress Controller instances running in the cluster. Then use _kubectl_ to delete the CRDs: ```shell kubectl delete -f crds/ ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io "aplogconfs.appprotect.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "appolicies.appprotect.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "apusersigs.appprotect.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "apdoslogconfs.appprotectdos.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "apdospolicies.appprotectdos.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "dosprotectedresources.appprotectdos.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "dnsendpoints.externaldns.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "globalconfigurations.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "policies.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "transportservers.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "virtualserverroutes.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "virtualservers.k8s.nginx.org" deleted ``` **Note:** If you run multiple NGINX Ingress Controller releases in your cluster with custom resources enabled, every release will share a single version of the CRDs. When uninstalling a release, ensure that you don’t remove the CRDs until there are no other NGINX Ingress Controller releases running in the cluster. The [Run multiple NGINX Ingress Controllers](/nic/install/multiple-controllers.md) topic has more details. ## Remove secrets If your deployment used NGINX Plus, you should also remove the secrets created for your license and the F5 registry. ```shell kubectl delete secret nplus-license ``` #### Example output ```text secret "nplus-license" deleted ``` ```shell kubectl delete secret regcred ``` #### Example output ```text secret "regcred" deleted ``` -------------------------------------------------------------------------------- title: "Use Helm to uninstall NGINX Ingress Controller LTS" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/helm/uninstall/index.md -------------------------------------------------------------------------------- This page describes how to use Helm to uninstall F5 NGINX Ingress Controller LTS. It explains how to remove the chart, then remove the custom resource definitions (CRDs). By following these instructions, you will remove NGINX Ingress Controller LTS from your Kubernetes cluster. ## Remove the Helm chart To uninstall NGINX Ingress Controller LTS, you must first remove the chart. To remove a release named **\**, use the following command: ```shell helm uninstall ``` The command removes all the Kubernetes components associated with the release, then deletes the release itself. ## Remove the CRDs After removing the release, pull the chart sources: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress --untar --version ``` **warning:** The next command will delete all corresponding custom resources in your cluster across all namespaces. Before using it, check there are no custom resources that you want to keep, and that there are no other NGINX Ingress Controller LTS instances running in the cluster. Then use _kubectl_ to delete the CRDs: ```shell kubectl delete -f crds/ ``` #### Example output ```text customresourcedefinition.apiextensions.k8s.io "aplogconfs.appprotect.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "appolicies.appprotect.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "apusersigs.appprotect.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "apdoslogconfs.appprotectdos.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "apdospolicies.appprotectdos.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "dosprotectedresources.appprotectdos.f5.com" deleted customresourcedefinition.apiextensions.k8s.io "dnsendpoints.externaldns.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "globalconfigurations.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "policies.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "transportservers.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "virtualserverroutes.k8s.nginx.org" deleted customresourcedefinition.apiextensions.k8s.io "virtualservers.k8s.nginx.org" deleted ``` **caution:** If you run multiple NGINX Ingress Controller LTS releases in your cluster with custom resources enabled, every release will share a single version of the CRDs. When uninstalling a release, ensure that you don’t remove the CRDs until there are no other NGINX Ingress Controller LTS releases running in the cluster. The [Run multiple NGINX Ingress Controller LTSs](/nic/lts/install/multiple-controllers.md) topic has more details. ## Remove secrets If your deployment used NGINX Plus, you should also remove the secrets created for your license and the F5 registry. ```shell kubectl delete secret nplus-license ``` #### Example output ```text secret "nplus-license" deleted ``` ```shell kubectl delete secret regcred ``` #### Example output ```text secret "regcred" deleted ``` -------------------------------------------------------------------------------- title: "Cross-namespace configuration" url: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/index.md -------------------------------------------------------------------------------- This topic explains how to spread Ingress configuration across different namespaces in F5 NGINX Ingress Controller. You can spread the Ingress configuration for a common host across multiple Ingress resources using Mergeable Ingress resources. Such resources can belong to the *same* or *different* namespaces. This enables easier management when using a large number of paths. See the [Mergeable Ingress Resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/mergeable-ingress-types) example in our GitHub repo. When [`-allow-empty-ingress-host`](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-allow-empty-ingress-host) is enabled, this also applies to Ingress resources without a `host`. As an alternative to Mergeable Ingress resources, you can use [VirtualServer and VirtualServerRoute resources](/nic//configuration/virtualserver-and-virtualserverroute-resources.md) for cross-namespace configuration. See the [Cross-Namespace Configuration](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/cross-namespace-configuration) example in our GitHub repo. -------------------------------------------------------------------------------- title: "Cross-namespace configuration" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/ingress-resources/cross-namespace-configuration/index.md -------------------------------------------------------------------------------- This topic explains how to spread Ingress configuration across different namespaces in F5 NGINX Ingress Controller LTS. You can spread the Ingress configuration for a common host across multiple Ingress resources using Mergeable Ingress resources. Such resources can belong to the *same* or *different* namespaces. This enables easier management when using a large number of paths. See the [Mergeable Ingress Resources](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/ingress-resources/mergeable-ingress-types) example in our GitHub repo. As an alternative to Mergeable Ingress resources, you can use [VirtualServer and VirtualServerRoute resources](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md) for cross-namespace configuration. See the [Cross-Namespace Configuration](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/custom-resources/cross-namespace-configuration) example in our GitHub repo. -------------------------------------------------------------------------------- title: "Custom templates" url: https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/custom-templates/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller uses templates to generate NGINX configuration for Ingress resources, VirtualServer resources and the main NGINX configuration file. You can customize the templates and apply them via the ConfigMap. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/custom-templates). -------------------------------------------------------------------------------- title: "Custom templates" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/global-configuration/custom-templates/index.md -------------------------------------------------------------------------------- F5 NGINX Ingress Controller LTS uses templates to generate NGINX configuration for Ingress resources, VirtualServer resources and the main NGINX configuration file. You can customize the templates and apply them via the ConfigMap. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/release-2026-lts/examples/shared-examples/custom-templates). -------------------------------------------------------------------------------- title: "Install NGINX Ingress Controller LTS with Manifests" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/manifests/index.md -------------------------------------------------------------------------------- This guide explains how to use Manifests to install F5 NGINX Ingress Controller LTS, then create both common and custom resources and set up role-based access control. ## Before you begin If you are using NGINX Plus, get the NGINX Ingress Controller LTS JWT and [create a license secret](/nic/lts/install/license-secret.md). ### Get the NGINX Controller Image **note:** We recommend installing the latest LTS patch release of NGINX Ingress Controller LTS, available on the GitHub repository's [releases page](https://github.com/nginx/kubernetes-ingress/releases). Choose one of the following methods to get the NGINX Ingress Controller LTS image: - **NGINX Plus Ingress Controller**: You have two options for this, both requiring an NGINX Ingress Controller LTS subscription. - - [Download NGINX Ingress Controller LTS from the F5 Registry](/nic/lts/install/images/registry-download.md) topic. - - [Add an NGINX Ingress Controller LTS image to your cluster](/nic/lts/install/images/add-image-to-cluster.md) ### Clone the repository Clone the NGINX Ingress Controller LTS repository using the command shown below, and replace `` with the specific release you want to use. ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch ``` For example, if you want to use version , the command would be: ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch v ``` This guide assumes you are using the latest release. Change the active directory. ```shell cd kubernetes-ingress ``` ## Set up role-based access control (RBAC) {#configure-rbac} **Note:** To complete these steps you need admin access to your cluster. Refer to your Kubernetes platform's documentation to set up admin access. For Google Kubernetes Engine (GKE), you can refer to their [Role-Based Access Control guide](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control). 1. Create a namespace and a service account: ```shell kubectl apply -f deployments/common/ns-and-sa.yaml ``` 2. Create a cluster role and binding for the service account: ```shell kubectl apply -f deployments/rbac/rbac.yaml ```
If you're planning to use F5 WAF for NGINX or F5 DoS for NGINX, additional roles and bindings are needed. 1. (F5 WAF for NGINX only) Create the *App Protect* role and binding: ```shell kubectl apply -f deployments/rbac/ap-rbac.yaml ``` 2. (F5 DoS for NGINX only) Create the *App Protect DoS* role and binding: ```shell kubectl apply -f deployments/rbac/apdos-rbac.yaml ``` ## Create common resources {#create-common-resources} In this section, you'll create resources that most NGINX Ingress Controller installations require: 1. (Optional) Create a secret for the default NGINX server's TLS certificate and key. Complete this step only if you're using the [default server TLS secret](/nic/configuration/global-configuration/command-line-arguments#cmdoption-default-server-tls-secret.md) command-line argument. If you're not, feel free to skip this step. By default, the server returns a _404 Not Found_ page for all requests when no ingress rules are set up. Although we provide a self-signed certificate and key for testing purposes, we recommend using your own certificate. ```shell make secrets kubectl apply -f examples/shared-examples/default-server-secret/default-server-secret.yaml ``` 2. Create a ConfigMap to customize your NGINX settings: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` 3. Create an `IngressClass` resource. NGINX Ingress Controller won't start without an `IngressClass` resource. ```shell kubectl apply -f deployments/common/ingress-class.yaml ``` If you want to make this NGINX Ingress Controller instance your cluster's default, uncomment the `ingressclass.kubernetes.io/is-default-class` annotation. This action will auto-assign `IngressClass` to new ingresses that don't specify an `ingressClassName`. ## Deploy NGINX Ingress Controller LTS {#deploy-ingress-controller} You have three options for deploying NGINX Ingress Controller LTS: - **Deployment**. Choose this method for the flexibility to dynamically change the number of NGINX Ingress Controller LTS replicas. - **DaemonSet**. Choose this method if you want NGINX Ingress Controller LTS to run on all nodes or a subset of nodes. - **StatefulSet**. Choose this method when you need stable, persistent storage and ordered deployment/scaling for your NGINX Ingress Controller LTS pods. Before you start, update the [command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments.md) for the NGINX Ingress Controller LTS container in the relevant manifest file to meet your specific requirements. ### Using a Deployment For additional context on managing containers using Kubernetes Deployments, refer to the official Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) documentation. When you deploy NGINX Ingress Controller as a Deployment, Kubernetes automatically sets up a single NGINX Ingress Controller pod. - For NGINX, run: ```shell kubectl apply -f deployments/deployment/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/deployment/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a DaemonSet For additional context on managing containers using Kubernetes DaemonSets, refer to the official Kubernetes [DaemonSets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) documentation. When you deploy NGINX Ingress Controller as a DaemonSet, Kubernetes creates an Ingress Controller pod on every node in the cluster. - For NGINX, run: ```shell kubectl apply -f deployments/daemon-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/daemon-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a StatefulSet For additional context on managing containers using Kubernetes StatefulSets, refer to the official Kubernetes [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) documentation. When you deploy NGINX Ingress Controller as a StatefulSet, Kubernetes creates pods with stable network identities and persistent storage. - For NGINX, run: ```shell kubectl apply -f deployments/stateful-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/stateful-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. **Note:** StatefulSets include persistent volume claims for nginx cache storage via `volumeClaimTemplates`. You may need to configure a StorageClass in your cluster or modify the volumeClaimTemplates section in the manifest to match your storage requirements. Other volumes (like those needed for App Protect modules) are configured in the regular `volumes` section, not in volumeClaimTemplates. ## Confirm NGINX Ingress Controller LTS is running To confirm the NGINX Ingress Controller pods are operational, run: ```shell kubectl get pods --namespace=nginx-ingress ``` ## How to access NGINX Ingress Controller LTS ### Using a Deployment or StatefulSet For Deployments and StatefulSets, you have two options for accessing NGINX Ingress Controller LTS pods. #### Option 1: Create a NodePort service For more information about the _NodePort_ service, refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport). 1. To create a service of type *NodePort*, run: ```shell kubectl create -f deployments/service/nodeport.yaml ``` Kubernetes automatically allocates two ports on every node in the cluster. You can access NGINX Ingress Controller LTS by combining any node's IP address with these ports. #### Option 2: Create a LoadBalancer service For more information about the _LoadBalancer_ service, refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer). 1. To set up a _LoadBalancer_ service, run one of the following commands based on your cloud provider: - GCP or Azure: ```shell kubectl apply -f deployments/service/loadbalancer.yaml ``` - AWS: ```shell kubectl apply -f deployments/service/loadbalancer-aws-elb.yaml ``` For more details see service guide [here](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/#annotations) 2. AWS users: Follow these additional steps to work with ELB in TCP mode. - Add the following keys to the `nginx-config.yaml` ConfigMap file, which you created in the [Create common resources](#create-common-resources) section. ```yaml proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" ``` - Update the ConfigMap: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` **note:** AWS users have more customization options for their load balancers. These include choosing the load balancer type and configuring SSL termination. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer) to learn more. 3. To access NGINX Ingress Controller LTS, get the public IP of your load balancer. - For GCP or Azure, run: ```shell kubectl get svc nginx-ingress --namespace=nginx-ingress ``` - For AWS find the DNS name: ```shell kubectl describe svc nginx-ingress --namespace=nginx-ingress ``` Resolve the DNS name into an IP address using `nslookup`: ```shell nslookup ``` You can also find more details about the public IP in the status section of an ingress resource. For more details, refer to the [Reporting Resources Status doc](/nic/lts/configuration/global-configuration/reporting-resources-status.md). ### Using a DaemonSet Connect to ports 80 and 443 using the IP address of any node in the cluster where NGINX Ingress Controller LTS is running. ## Uninstall NGINX Ingress Controller LTS **warning:** Proceed with caution when performing these steps, as they will remove NGINX Ingress Controller LTS and all related resources, potentially affecting your running services. 1. **Delete the nginx-ingress namespace**: To remove NGINX Ingress Controller LTS and all auxiliary resources, run: ```shell kubectl delete namespace nginx-ingress ``` 1. **Remove the cluster role and cluster role binding**: ```shell kubectl delete clusterrole nginx-ingress kubectl delete clusterrolebinding nginx-ingress ``` 1. **Delete the Custom Resource Definitions**: #### Deleting CRDs from single YAML Delete core custom resource definitions: ```shell kubectl delete -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` #### Deleting CRDs after cloning the repo Delete core custom resource definitions: ```shell kubectl delete -f config/crd/bases/crds.yaml ``` -------------------------------------------------------------------------------- title: "Install NGINX Ingress Controller with Manifests" url: https://docs.nginx.com/nginx-ingress-controller/install/manifests/index.md -------------------------------------------------------------------------------- This guide explains how to use Manifests to install F5 NGINX Ingress Controller, then create both common and custom resources and set up role-based access control. ## Before you begin If you are using NGINX Plus, get the NGINX Ingress Controller JWT and [create a license secret](/nic/install/license-secret.md). ### Get the NGINX Controller Image **Note:** We recommend the most recent stable version of NGINX Ingress Controller, available on the GitHub repository's [releases page](https://github.com/nginx/kubernetes-ingress/releases). Choose one of the following methods to get the NGINX Ingress Controller image: - **NGINX Ingress Controller**: Download the image `nginx/nginx-ingress` from [DockerHub](https://hub.docker.com/r/nginx/nginx-ingress). - **NGINX Plus Ingress Controller**: You have two options for this, both requiring an NGINX Ingress Controller subscription. - - [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) topic. - - [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) - **Build your own image**: To build your own image, follow the [Build NGINX Ingress Controller](/nic/install/build.md) topic. ### Clone the repository Clone the NGINX Ingress Controller repository using the command shown below, and replace `` with the specific release you want to use. ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch ``` For example, if you want to use version , the command would be: ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch v ``` This guide assumes you are using the latest release. Change the active directory. ```shell cd kubernetes-ingress ``` ### App Protect DoS To use App Protect DoS, install the App Protect DoS Arbitrator using the provided manifests in the same namespace as the NGINX Ingress Controller. If you install multiple NGINX Ingress Controllers in the same namespace, they will need to share the same Arbitrator because there can only be one Arbitrator in a single namespace. ## Set up role-based access control (RBAC) {#configure-rbac} **Note:** To complete these steps you need admin access to your cluster. Refer to your Kubernetes platform's documentation to set up admin access. For Google Kubernetes Engine (GKE), you can refer to their [Role-Based Access Control guide](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control). 1. Create a namespace and a service account: ```shell kubectl apply -f deployments/common/ns-and-sa.yaml ``` 2. Create a cluster role and binding for the service account: ```shell kubectl apply -f deployments/rbac/rbac.yaml ```
If you're planning to use F5 WAF for NGINX or F5 DoS for NGINX, additional roles and bindings are needed. 1. (F5 WAF for NGINX only) Create the *App Protect* role and binding: ```shell kubectl apply -f deployments/rbac/ap-rbac.yaml ``` 2. (F5 DoS for NGINX only) Create the *App Protect DoS* role and binding: ```shell kubectl apply -f deployments/rbac/apdos-rbac.yaml ``` ## Create common resources {#create-common-resources} In this section, you'll create resources that most NGINX Ingress Controller installations require: 1. (Optional) Create a secret for the default NGINX server's TLS certificate and key. Complete this step only if you're using the [default server TLS secret](/nic/configuration/global-configuration/command-line-arguments#cmdoption-default-server-tls-secret.md) command-line argument. If you're not, feel free to skip this step. By default, the server returns a _404 Not Found_ page for all requests when no ingress rules are set up. Although we provide a self-signed certificate and key for testing purposes, we recommend using your own certificate. ```shell make secrets kubectl apply -f examples/shared-examples/default-server-secret/default-server-secret.yaml ``` 2. Create a ConfigMap to customize your NGINX settings: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` 3. Create an `IngressClass` resource. NGINX Ingress Controller won't start without an `IngressClass` resource. ```shell kubectl apply -f deployments/common/ingress-class.yaml ``` If you want to make this NGINX Ingress Controller instance your cluster's default, uncomment the `ingressclass.kubernetes.io/is-default-class` annotation. This action will auto-assign `IngressClass` to new ingresses that don't specify an `ingressClassName`. ## Create core custom resources {#create-custom-resources} To make sure your NGINX Ingress Controller pods reach the `Ready` state, you'll need to create custom resource definitions (CRDs) for various components. Alternatively, you can disable this requirement by setting the `-enable-custom-resources` command-line argument to `false`. There are two ways you can install the custom resource definitions: 1. Using a URL to apply a single CRD yaml file, which we recommend. 1. Applying your local copy of the CRD yaml files, which requires you to clone the repository. The core custom CRDs are the following: - [VirtualServer and VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) - [TransportServer](/nic/configuration/transportserver-resource.md) - [Policy](/nic/configuration/policy-resource/before-you-begin.md) - [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) #### Install CRDs from single YAML ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` #### Install CRDs after cloning the repo **Note:** Read the steps outlined in [Upgrade from 3.x to 4.x](/nic/install/upgrade.md#upgrade-from-3x-to-4x) before running the CRD upgrade and perform the steps if applicable. ```shell kubectl apply -f config/crd/bases/k8s.nginx.org_virtualservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_virtualserverroutes.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_transportservers.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_policies.yaml kubectl apply -f config/crd/bases/k8s.nginx.org_globalconfigurations.yaml ``` ### Create optional custom resources There are optional CRDs that are necessary if you want to use F5 WAF for NGINX or F5 DoS for NGINX. _F5 WAF for NGINX_: - `APPolicy` - `APLogConf` - `APUserSig` _F5 DoS for NGINX_: - `APDosPolicy` - `APDosLogConf` - `DosProtectedResource` #### Install CRDs from single YAML _F5 WAF for NGINX_: **Note:** This step can be skipped if you are using F5 WAF for NGINX module with policy bundles. ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds-nap-waf.yaml ``` _F5 DoS for NGINX_: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds-nap-dos.yaml ``` #### Install CRDs after cloning the repo _F5 WAF for NGINX_: **Note:** This step can be skipped if you are using F5 WAF for NGINX module with policy bundles. ```shell kubectl apply -f config/crd/bases/appprotect.f5.com_aplogconfs.yaml kubectl apply -f config/crd/bases/appprotect.f5.com_appolicies.yaml kubectl apply -f config/crd/bases/appprotect.f5.com_apusersigs.yaml ``` _F5 DoS for NGINX_: ```shell kubectl apply -f config/crd/bases/appprotectdos.f5.com_apdoslogconfs.yaml kubectl apply -f config/crd/bases/appprotectdos.f5.com_apdospolicy.yaml kubectl apply -f config/crd/bases/appprotectdos.f5.com_dosprotectedresources.yaml ``` ## Deploy NGINX Ingress Controller {#deploy-ingress-controller} You have three options for deploying NGINX Ingress Controller: - **Deployment**. Choose this method for the flexibility to dynamically change the number of NGINX Ingress Controller replicas. - **DaemonSet**. Choose this method if you want NGINX Ingress Controller to run on all nodes or a subset of nodes. - **StatefulSet**. Choose this method when you need stable, persistent storage and ordered deployment/scaling for your NGINX Ingress Controller pods. Before you start, update the [command-line arguments](/nic/configuration/global-configuration/command-line-arguments.md) for the NGINX Ingress Controller container in the relevant manifest file to meet your specific requirements. ### Using a Deployment For additional context on managing containers using Kubernetes Deployments, refer to the official Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) documentation. When you deploy NGINX Ingress Controller as a Deployment, Kubernetes automatically sets up a single NGINX Ingress Controller pod. - For NGINX, run: ```shell kubectl apply -f deployments/deployment/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/deployment/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a DaemonSet For additional context on managing containers using Kubernetes DaemonSets, refer to the official Kubernetes [DaemonSets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) documentation. When you deploy NGINX Ingress Controller as a DaemonSet, Kubernetes creates an Ingress Controller pod on every node in the cluster. - For NGINX, run: ```shell kubectl apply -f deployments/daemon-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/daemon-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. ### Using a StatefulSet For additional context on managing containers using Kubernetes StatefulSets, refer to the official Kubernetes [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) documentation. When you deploy NGINX Ingress Controller as a StatefulSet, Kubernetes creates pods with stable network identities and persistent storage. - For NGINX, run: ```shell kubectl apply -f deployments/stateful-set/nginx-ingress.yaml ``` - For NGINX Plus, run: ```shell kubectl apply -f deployments/stateful-set/nginx-plus-ingress.yaml ``` Update the `nginx-plus-ingress.yaml` file to include your chosen image from the F5 Container registry or your custom container image. **Note:** StatefulSets include persistent volume claims for nginx cache storage via `volumeClaimTemplates`. You may need to configure a StorageClass in your cluster or modify the volumeClaimTemplates section in the manifest to match your storage requirements. Other volumes (like those needed for App Protect modules) are configured in the regular `volumes` section, not in volumeClaimTemplates. ## Confirm NGINX Ingress Controller is running To confirm the NGINX Ingress Controller pods are operational, run: ```shell kubectl get pods --namespace=nginx-ingress ``` ## How to access NGINX Ingress Controller ### Using a Deployment or StatefulSet For Deployments and StatefulSets, you have two options for accessing NGINX Ingress Controller pods. #### Option 1: Create a NodePort service For more information about the _NodePort_ service, refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport). 1. To create a service of type *NodePort*, run: ```shell kubectl create -f deployments/service/nodeport.yaml ``` Kubernetes automatically allocates two ports on every node in the cluster. You can access NGINX Ingress Controller by combining any node's IP address with these ports. #### Option 2: Create a LoadBalancer service For more information about the _LoadBalancer_ service, refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer). 1. To set up a _LoadBalancer_ service, run one of the following commands based on your cloud provider: - GCP or Azure: ```shell kubectl apply -f deployments/service/loadbalancer.yaml ``` - AWS: ```shell kubectl apply -f deployments/service/loadbalancer-aws-elb.yaml ``` For more details see service guide [here](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/#annotations) 2. AWS users: Follow these additional steps to work with ELB in TCP mode. - Add the following keys to the `nginx-config.yaml` ConfigMap file, which you created in the [Create common resources](#create-common-resources) section. ```yaml proxy-protocol: "True" real-ip-header: "proxy_protocol" set-real-ip-from: "0.0.0.0/0" ``` - Update the ConfigMap: ```shell kubectl apply -f deployments/common/nginx-config.yaml ``` **Note:** AWS users have more customization options for their load balancers. These include choosing the load balancer type and configuring SSL termination. Refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer) to learn more. 3. To access NGINX Ingress Controller, get the public IP of your load balancer. - For GCP or Azure, run: ```shell kubectl get svc nginx-ingress --namespace=nginx-ingress ``` - For AWS find the DNS name: ```shell kubectl describe svc nginx-ingress --namespace=nginx-ingress ``` Resolve the DNS name into an IP address using `nslookup`: ```shell nslookup ``` You can also find more details about the public IP in the status section of an ingress resource. For more details, refer to the [Reporting Resources Status doc](/nic/configuration/global-configuration/reporting-resources-status.md). ### Using a DaemonSet Connect to ports 80 and 443 using the IP address of any node in the cluster where NGINX Ingress Controller is running. ## Uninstall NGINX Ingress Controller **Note:** Proceed with caution when performing these steps, as they will remove NGINX Ingress Controller and all related resources, potentially affecting your running services. 1. **Delete the nginx-ingress namespace**: To remove NGINX Ingress Controller and all auxiliary resources, run: ```shell kubectl delete namespace nginx-ingress ``` 1. **Remove the cluster role and cluster role binding**: ```shell kubectl delete clusterrole nginx-ingress kubectl delete clusterrolebinding nginx-ingress ``` 1. **Delete the Custom Resource Definitions**: #### Deleting CRDs from single YAML Delete core custom resource definitions: ```shell kubectl delete -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` Delete custom resource definitions for the F5 WAF for NGINX module: ```shell kubectl delete -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds-nap-waf.yaml ``` Delete custom resource definitions for the F5 DoS for NGINX module: ```shell kubectl delete -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds-nap-dos.yaml ``` #### Deleting CRDs after cloning the repo Delete core custom resource definitions: ```shell kubectl delete -f config/crd/bases/crds.yaml ``` Delete custom resource definitions for the F5 WAF for NGINX module: ```shell kubectl delete -f config/crd/bases/crds-nap-waf.yaml ``` Delete custom resource definitions for the F5 DoS for NGINX module: ```shell kubectl delete -f config/crd/bases/crds-nap-dos.yaml ``` -------------------------------------------------------------------------------- title: "Policy precedence and override rules" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/precedence-and-overrides/index.md -------------------------------------------------------------------------------- When more than one policy of the same type applies to a resource, NGINX Ingress Controller uses precedence rules to decide which policy takes effect. ## VirtualServer and VirtualServerRoute Policy precedence goes from broader scope to narrower scope: - `VirtualServer.spec.policies` - `VirtualServer.route.policies` - `VirtualServerRoute.subroute.policies` If the same policy type appears at multiple levels, the more specific level wins. For example: - Route-level `accessControl` overrides spec-level `accessControl`. - Subroute-level `cors` overrides route-level `cors`. ## Ingress and mergeable Ingress For Ingress: - Policies apply to the whole Ingress. - With mergeable Ingress, minion policies override master policies of the same type. ## What's next See the [policy type reference](/nic/configuration/policy-resource/policy-reference.md) for the fields and merging behavior of each policy type. -------------------------------------------------------------------------------- title: "Product telemetry" url: https://docs.nginx.com/nginx-ingress-controller/lts/overview/product-telemetry/index.md -------------------------------------------------------------------------------- Learn why, what and how F5 NGINX Ingress Controller LTS collects telemetry. --- ## Overview NGINX Ingress Controller LTS collects product telemetry data to allow its developers to understand how it's deployed and configured by users. This data is used to triage development work, prioritizing features and functionality that will benefit the most people. Product telemetry is enabled by default, collected once every 24 hours. It's then sent over HTTPS to a service managed by F5 at `oss.edge.df.f5.com`. **note:** If you would prefer not to send any telemetry data, you can [opt-out](#opt-out) when installing NGINX Ingress Controller LTS. --- ## Data collected These are the data points collected and reported by NGINX Ingress Controller LTS: - **Project Name** The name of the software, which will be labelled `NIC`. - **Project Version** NGINX Ingress Controller LTS version. - **Project Architecture** The architecture of the kubernetes environment. (e.g. amd64, arm64, etc...) - **Cluster ID** A unique identifier of the kubernetes cluster that NGINX Ingress Controller LTS is deployed to. - **Cluster Version** The version of the Kubernetes cluster. - **Cluster Platform** The platform that the kubernetes cluster is operating on. (e.g. eks, aks, etc...) - **Cluster Node Count** The number of nodes in the cluster that NGINX Ingress Controller LTS is deployed to. - **Installation ID** Used to identify a unique installation of NGINX Ingress Controller LTS. - **VirtualServers** The number of VirtualServer resources managed by NGINX Ingress Controller LTS. - **VirtualServerRoutes** The number of VirtualServerRoute resources managed by NGINX Ingress Controller LTS. - **TransportServers** The number of TransportServer resources managed by NGINX Ingress Controller LTS. - **Replicas** Number of Deployment or StatefulSet replicas, or DaemonSet instances. - **Secrets** Number of Secret resources managed by NGINX Ingress Controller LTS. - **ClusterIPServices** Number of ClusterIP Services managed by NGINX Ingress Controller LTS. - **NodePortServices** Number of NodePort Services managed by NGINX Ingress Controller LTS. - **LoadBalancerServices** Number of LoadBalancer Services managed by NGINX Ingress Controller LTS. - **ExternalNameServices** Number of ExternalName Services managed by NGINX Ingress Controller LTS. - **RegularIngressCount** The number of Regular Ingress resources managed by NGINX Ingress Controller LTS. - **MasterIngressCount** The number of Master Ingress resources managed by NGINX Ingress Controller LTS. - **MinionIngressCount** The number of Minion Ingress resources managed by NGINX Ingress Controller LTS. - **IngressClasses** Number of Ingress Classes in the cluster. - **IngressAnnotations** List of Ingress annotations managed by NGINX Ingress Controller LTS - **AccessControlPolicies** Number of AccessControl policies. - **RateLimitPolicies** Number of RateLimit policies. - **APIKeyPolicies** Number of API Key Auth policies. - **JWTAuthPolicies** Number of JWTAuth policies. - **BasicAuthPolicies** Number of BasicAuth policies. - **IngressMTLSPolicies** Number of IngressMTLS policies. - **EgressMTLSPolicies** Number of EgressMTLS policies. - **OIDCPolicies** Number of OIDC policies. - **WAFPolicies** Number of WAF policies. - **CachePolicies** Number of Cache policies. - **CORSPolicies** Number of CORS policies. - **GlobalConfiguration** Represents the use of a GlobalConfiguration resource. - **AppProtectVersion** The AppProtect version - **IsPlus** Represents whether NGINX is Plus or OSS - **InstallationFlags** List of command line arguments configured for NGINX Ingress Controller LTS - **BuildOS** The base operating system image in which NGINX Ingress Controller LTS is running on. - **ConfigMapKeys** List of ConfigMap keys from the main ConfigMap configured for NGINX Ingress Controller LTS. - **MGMTConfigMapKeys** List of MGMTConfigMap keys from the mgmt ConfigMap configured for NGINX Ingress Controller LTS. --- ## Opt out Product telemetry can be disabled when installing NGINX Ingress Controller LTS. ### Helm When installing or upgrading NGINX Ingress Controller LTS with Helm, set the `controller.telemetryReporting.enable` option to `false`. This can be set directly in the `values.yaml` file, or using the `--set` option ```shell helm upgrade --install ... --set controller.telemetryReporting.enable=false ``` --- ### Manifests When installing NGINX Ingress Controller LTS with Manifests, set the `-enable-telemetry-reporting` flag to `false` -------------------------------------------------------------------------------- title: "Product telemetry" url: https://docs.nginx.com/nginx-ingress-controller/overview/product-telemetry/index.md -------------------------------------------------------------------------------- Learn why, what and how F5 NGINX Ingress Controller collects telemetry. --- ## Overview NGINX Ingress Controller collects product telemetry data to allow its developers to understand how it's deployed and configured by users. This data is used to triage development work, prioritizing features and functionality that will benefit the most people. Product telemetry is enabled by default, collected once every 24 hours. It's then sent over HTTPS to a service managed by F5 at `oss.edge.df.f5.com`. **Note:** If you would prefer not to send any telemetry data, you can [opt-out](#opt-out) when installing NGINX Ingress Controller. --- ## Data collected These are the data points collected and reported by NGINX Ingress Controller: - **Project Name** The name of the software, which will be labelled `NIC`. - **Project Version** NGINX Ingress Controller version. - **Project Architecture** The architecture of the kubernetes environment. (e.g. amd64, arm64, etc...) - **Cluster ID** A unique identifier of the kubernetes cluster that NGINX Ingress Controller is deployed to. - **Cluster Version** The version of the Kubernetes cluster. - **Cluster Platform** The platform that the kubernetes cluster is operating on. (e.g. eks, aks, etc...) - **Cluster Node Count** The number of nodes in the cluster that NGINX Ingress Controller is deployed to. - **Installation ID** Used to identify a unique installation of NGINX Ingress Controller. - **VirtualServers** The number of VirtualServer resources managed by NGINX Ingress Controller. - **VirtualServerRoutes** The number of VirtualServerRoute resources managed by NGINX Ingress Controller. - **TransportServers** The number of TransportServer resources managed by NGINX Ingress Controller. - **Replicas** Number of Deployment or StatefulSet replicas, or DaemonSet instances. - **Secrets** Number of Secret resources managed by NGINX Ingress Controller. - **ClusterIPServices** Number of ClusterIP Services managed by NGINX Ingress Controller. - **NodePortServices** Number of NodePort Services managed by NGINX Ingress Controller. - **LoadBalancerServices** Number of LoadBalancer Services managed by NGINX Ingress Controller. - **ExternalNameServices** Number of ExternalName Services managed by NGINX Ingress Controller. - **RegularIngressCount** The number of Regular Ingress resources managed by NGINX Ingress Controller. - **MasterIngressCount** The number of Master Ingress resources managed by NGINX Ingress Controller. - **MinionIngressCount** The number of Minion Ingress resources managed by NGINX Ingress Controller. - **IngressClasses** Number of Ingress Classes in the cluster. - **IngressAnnotations** List of Ingress annotations managed by NGINX Ingress Controller - **AccessControlPolicies** Number of AccessControl policies. - **RateLimitPolicies** Number of RateLimit policies. - **APIKeyPolicies** Number of API Key Auth policies. - **JWTAuthPolicies** Number of JWTAuth policies. - **BasicAuthPolicies** Number of BasicAuth policies. - **IngressMTLSPolicies** Number of IngressMTLS policies. - **EgressMTLSPolicies** Number of EgressMTLS policies. - **OIDCPolicies** Number of OIDC policies. - **WAFPolicies** Number of WAF policies. - **CachePolicies** Number of Cache policies. - **CORSPolicies** Number of CORS policies. - **GlobalConfiguration** Represents the use of a GlobalConfiguration resource. - **AppProtectVersion** The AppProtect version - **IsPlus** Represents whether NGINX is Plus or OSS - **InstallationFlags** List of command line arguments configured for NGINX Ingress Controller - **BuildOS** The base operating system image in which NGINX Ingress Controller is running on. - **ConfigMapKeys** List of ConfigMap keys from the main ConfigMap configured for NGINX Ingress Controller. - **MGMTConfigMapKeys** List of MGMTConfigMap keys from the mgmt ConfigMap configured for NGINX Ingress Controller. --- ## Opt out Product telemetry can be disabled when installing NGINX Ingress Controller. ### Helm When installing or upgrading NGINX Ingress Controller with Helm, set the `controller.telemetryReporting.enable` option to `false`. This can be set directly in the `values.yaml` file, or using the `--set` option ```shell helm upgrade --install ... --set controller.telemetryReporting.enable=false ``` --- ### Manifests When installing NGINX Ingress Controller with Manifests, set the `-enable-telemetry-reporting` flag to `false` -------------------------------------------------------------------------------- title: "Enable Service Insight" url: https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/service-insight/index.md -------------------------------------------------------------------------------- The Service Insight feature is available only for F5 NGINX Plus. The F5 NGINX Ingress Controller exposes an endpoint which provides host statistics for services exposed using the VirtualServer (VS) and TransportServer (TS) resources. It exposes data in JSON format and returns HTTP status codes. The response body holds information about the total, down and the unhealthy number of upstream pods associated with the configured hostname. Returned HTTP codes indicate the health of the service. The service is indicated as not healthy (HTTP response code different than 200 OK) if all upstreams (pods) are determined unhealthy by NGINX Plus. The service is healthy if at least one upstream pod is healthy as determined by NGINX Plus. In this case, the endpoint returns HTTP code 200 OK. NGINX Plus determination of healthy can be tuned using advanced health checks, and also dynamically relate to pods responses and responsiveness. See [Upstream Healthcheck](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#upstream) ## Enabling Service Insight Endpoint If you're using *Kubernetes manifests* (Deployment, DaemonSet, or StatefulSet) to install the Ingress Controller, to enable the Service Insight endpoint: 1. Run the Ingress Controller with the `-enable-service-insight` [command-line argument](/nic/configuration/global-configuration/command-line-arguments.md). This will expose the Ingress Controller endpoint via paths `/probe/{hostname}` for Virtual Servers, and `/probe/ts/{service_name}` for Transport Servers on port `9114` (customizable with the `-service-insight-listen-port` command-line argument). The `service_name` parameter refers to the name of the deployed service (the service specified under `upstreams` in the transport server). 1. To enable TLS for the Service Insight endpoint, configure the `-service-insight-tls-secret` cli argument with the namespace and name of a TLS Secret. 1. Add the Service Insight port to the list of the ports of the Ingress Controller container in the template of the Ingress Controller pod: ```yaml - name: service-insight containerPort: 9114 ``` If you're using *Helm* to install the Ingress Controller, to enable Service Insight endpoint, configure the `serviceInsight.*` parameters of the Helm chart. See the [Installation with Helm](/nic/install/helm.md) doc. ## Available Statistics and HTTP Response Codes The Service Insight provides the following statistics: - Total number of VS or TS pods - Number of VS or TS pods in 'Up' state - Number of VS or TS pods in 'Unhealthy' state These statistics are returned as JSON: ```json { "Total": , "Up": , "Unhealthy": } ``` Response codes: - HTTP 200 OK - Service is healthy - HTTP 404 Not Found - No upstreams/VS/TS found for the requested hostname/name - HTTP 418 I'm a teapot - The service is down (All upstreams/VS/TS are "Unhealthy") **Note**: wildcards in hostnames are not supported at the moment. -------------------------------------------------------------------------------- title: "Enable Service Insight" url: https://docs.nginx.com/nginx-ingress-controller/lts/logging-and-monitoring/service-insight/index.md -------------------------------------------------------------------------------- The F5 NGINX Ingress Controller LTS exposes an endpoint which provides host statistics for services exposed using the VirtualServer (VS) and TransportServer (TS) resources. It exposes data in JSON format and returns HTTP status codes. The response body holds information about the total, down and the unhealthy number of upstream pods associated with the configured hostname. Returned HTTP codes indicate the health of the service. The service is indicated as not healthy (HTTP response code different than 200 OK) if all upstreams (pods) are determined unhealthy by NGINX Plus. The service is healthy if at least one upstream pod is healthy as determined by NGINX Plus. In this case, the endpoint returns HTTP code 200 OK. NGINX Plus determination of healthy can be tuned using advanced health checks, and also dynamically relate to pods responses and responsiveness. See [Upstream Healthcheck](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#upstream) ## Enabling Service Insight Endpoint If you're using *Kubernetes manifests* (Deployment, DaemonSet, or StatefulSet) to install the Ingress Controller, to enable the Service Insight endpoint: 1. Run the Ingress Controller with the `-enable-service-insight` [command-line argument](/nic/lts/configuration/global-configuration/command-line-arguments.md). This will expose the Ingress Controller endpoint via paths `/probe/{hostname}` for Virtual Servers, and `/probe/ts/{service_name}` for Transport Servers on port `9114` (customizable with the `-service-insight-listen-port` command-line argument). The `service_name` parameter refers to the name of the deployed service (the service specified under `upstreams` in the transport server). 1. To enable TLS for the Service Insight endpoint, configure the `-service-insight-tls-secret` cli argument with the namespace and name of a TLS Secret. 1. Add the Service Insight port to the list of the ports of the Ingress Controller container in the template of the Ingress Controller pod: ```yaml - name: service-insight containerPort: 9114 ``` If you're using *Helm* to install the Ingress Controller, to enable Service Insight endpoint, configure the `serviceInsight.*` parameters of the Helm chart. See the [Installation with Helm](/nic/lts/install/helm.md) doc. ## Available Statistics and HTTP Response Codes The Service Insight provides the following statistics: - Total number of VS or TS pods - Number of VS or TS pods in 'Up' state - Number of VS or TS pods in 'Unhealthy' state These statistics are returned as JSON: ```json { "Total": , "Up": , "Unhealthy": } ``` Response codes: - HTTP 200 OK - Service is healthy - HTTP 404 Not Found - No upstreams/VS/TS found for the requested hostname/name - HTTP 418 I'm a teapot - The service is down (All upstreams/VS/TS are "Unhealthy") **Note**: wildcards in hostnames are not supported at the moment. -------------------------------------------------------------------------------- title: "Install NGINX Ingress Controller with NGINX Ingress Operator" url: https://docs.nginx.com/nginx-ingress-controller/install/operator/index.md -------------------------------------------------------------------------------- This document explains how to install F5 NGINX Ingress Controller using NGINX Ingress Operator. ## Before you begin If you're using NGINX Plus, get the NGINX Ingress Controller JWT and [create a license secret](/nic/install/license-secret.md). **Note:** We recommend the most recent stable version of NGINX Ingress Controller, available on the GitHub repository's [releases page](https://github.com/nginx/kubernetes-ingress/releases). Make sure you have access to the NGINX Ingress Controller image: - For NGINX Ingress Controller, use the image `nginx/nginx-ingress` from [DockerHub](https://hub.docker.com/r/nginx/nginx-ingress). - For NGINX Plus Ingress Controller, view the [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download) topic for details on how to pull the image from the F5 Docker registry. - The [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) topic describes how to use your subscription JWT token to get the image. - The [Build NGINX Ingress Controller](/nic/install/build.md) topic explains how to push an image to a private Docker registry. Install the NGINX Ingress Operator following the [instructions](https://github.com/nginx/nginx-ingress-helm-operator/blob/main/docs/installation.md). Create the SecurityContextConstraint as outlined in the ["Getting Started" instructions](https://github.com/nginx/nginx-ingress-helm-operator/blob/main/README.md#getting-started). **Note:** If you're upgrading your operator installation to a later release, navigate [here](https://github.com/nginx/nginx-ingress-helm-operator/blob/main/helm-charts/nginx-ingress) and run `kubectl apply -f crds/` or `oc apply -f crds/` as a prerequisite ## Create the NGINX Ingress Controller manifest Create a manifest `nginx-ingress-controller.yaml` with the following content: ```yaml apiVersion: charts.nginx.org/v1alpha1 kind: NginxIngress metadata: name: nginxingress-sample namespace: nginx-ingress spec: controller: image: pullPolicy: IfNotPresent repository: nginx/nginx-ingress tag: -ubi ingressClass: name: nginx kind: deployment nginxplus: false replicaCount: 1 serviceAccount: imagePullSecretName: "" ``` **Note:** For NGINX Plus, change the `image.repository` and `image.tag` values and change `nginxPlus` to `True`. If required, set the `serviceAccount.imagePullSecretName` or `serviceAccount.imagePullSecretsNames` to the name of the pre-created docker config secret that should be associated with the ServiceAccount. ## Deploy NGINX Ingress Controller ```shell kubectl apply -f nginx-ingress-controller.yaml ``` A new instance of NGINX Ingress Controller will be deployed by the NGINX Ingress Operator in the `default` namespace with default parameters. To configure other parameters of the NginxIngressController resource, check the [documentation](https://github.com/nginx/nginx-ingress-helm-operator/blob/main/docs/nginx-ingress-controller.md). ## Troubleshooting If you experience an `OOMkilled` error when deploying the NGINX Ingress Operator in a large cluster, it's likely because the Helm operator is caching all Kubernetes objects and using up too much memory. If you encounter this issue, try the following solutions: - Set the operator to only watch one namespace. - If monitoring multiple namespaces is required, consider manually increasing the memory limit for the operator. Keep in mind that this value might be overwritten after a release update. We are working with the OpenShift team to resolve this issue. -------------------------------------------------------------------------------- title: "Policy type reference" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/policy-reference/index.md -------------------------------------------------------------------------------- This reference describes the fields and merging behavior for each Policy type. ## AccessControl The access control policy configures NGINX to deny or allow requests from clients with the specified IP addresses or subnets. For example, the following policy allows access for clients from the subnet `10.0.0.0/8` and denies access for any other clients: ```yaml accessControl: allow: - 10.0.0.0/8 ``` In contrast, the following policy does the opposite. It denies access for clients from `10.0.0.0/8` and allows access for any other clients: ```yaml accessControl: deny: - 10.0.0.0/8 ``` **Note:** This feature uses the NGINX [ngx_http_access_module](https://nginx.org/en/docs/http/ngx_http_access_module.html). The NGINX Ingress Controller access control policy supports either allow rules or deny rules, but not both, unlike the module itself. |Field | Description | Type | Required | | ---| ---| ---| --- | |``allow`` | Allows access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | |``deny`` | Denies access for the specified networks or addresses. For example, ``192.168.1.1`` or ``10.1.1.0/16``. | ``[]string`` | No | \* an accessControl must include either `allow` or `deny`. | ### AccessControl merging behavior A VirtualServer or VirtualServerRoute can reference multiple access control policies. For example, this configuration references two policies, each with a configured allow list: ```yaml policies: - name: allow-policy-one - name: allow-policy-two ``` When a resource references more than one access control policy, NGINX Ingress Controller merges the contents into a single allow list or a single deny list. NGINX Ingress Controller doesn't support referencing both allow and deny policies together, as shown in the following example. If a resource references both allow and deny lists, NGINX Ingress Controller uses only the allow list policies. ```yaml policies: - name: deny-policy - name: allow-policy-one - name: allow-policy-two ``` ## RateLimit The rate limit policy configures NGINX to limit the processing rate of requests. For example, the following policy limits all subsequent requests from a single IP address once the rate exceeds 10 requests per second: ```yaml rateLimit: rate: 10r/s zoneSize: 10M key: ${binary_remote_addr} ``` **Note:** This feature uses the NGINX [ngx_http_limit_req_module](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html). **Note:** When you turn on the [zone sync feature](/nic/configuration/global-configuration/configmap-resource.md#zone-sync) with NGINX Plus, NGINX Ingress Controller synchronizes the rate limiting zone across all replicas in the cluster. This means all replicas know about requests that other replicas in the cluster have already rate limited. |Field | Description | Type | Required | | ---| ---| ---| --- | |``rate`` | The rate of requests permitted. The rate is specified in requests per second (r/s) or requests per minute (r/m). | ``string`` | Yes | |``key`` | The key to which the rate limit is applied. Can contain text, variables, or a combination of them. Variables must be surrounded by ``${}``. For example: ``${binary_remote_addr}``. Accepted variables are ``$binary_remote_addr``, ``$request_uri``,``$request_method``, ``$url``, ``$http_``, ``$args``, ``$arg_``, ``$cookie_``, ``$jwt_claim_``. | ``string`` | Yes | |``zoneSize`` | Size of the shared memory zone. Only positive values are allowed. Allowed suffixes are ``k`` or ``m``, if none are present ``k`` is assumed. | ``string`` | Yes | |``delay`` | The delay parameter specifies a limit at which excessive requests become delayed. If not set all excessive requests are delayed. | ``int`` | No | |``noDelay`` | Disables the delaying of excessive requests while requests are being limited. Overrides ``delay`` if both are set. | ``bool`` | No | |``burst`` | Excessive requests are delayed until their number exceeds the ``burst`` size, in which case the request is terminated with an error. | ``int`` | No | |``dryRun`` | Turns on dry run mode. In this mode, NGINX Ingress Controller doesn't apply the rate limit, but it accounts for the number of excessive requests as usual in the shared memory zone. | ``bool`` | No | |``logLevel`` | Sets the desired logging level for cases when the server refuses to process requests due to rate exceeding, or delays request processing. Allowed values are ``info``, ``notice``, ``warn`` or ``error``. Default is ``error``. | ``string`` | No | |``rejectCode`` | Sets the status code to return in response to rejected requests. Must fall into the range ``400..599``. Default is ``503``. | ``int`` | No | |``scale`` | Keeps the rate limit constant by dividing the configured rate by the number of NGINX Ingress Controller pods currently serving traffic. This adjustment keeps the rate limit consistent, even as the number of pods fluctuates due to autoscaling. **This doesn't work correctly if requests from a client aren't distributed evenly across all Ingress Controller pods** (for example, with sticky sessions or long-lived TCP connections with many requests). In these cases, [zone sync](/nic/configuration/global-configuration/configmap-resource.md#zone-sync) gives better results. Turning on `zone-sync` suppresses this setting. | ``bool`` | No | |``condition`` | Add a condition to a rate-limit policy. | [ratelimit.condition](#ratelimitcondition) | No | **Note:** For each policy referenced in a VirtualServer or its VirtualServerRoutes, NGINX Ingress Controller generates a single rate limiting zone defined by the [`limit_req_zone`](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) directive. If two VirtualServer resources reference the same policy, NGINX Ingress Controller generates two different rate limiting zones, one zone per VirtualServer. ### RateLimit merging behavior A VirtualServer or VirtualServerRoute can reference multiple rate limit policies. For example, this configuration references two policies: ```yaml policies: - name: rate-limit-policy-one - name: rate-limit-policy-two ``` When a resource references more than one rate limit policy, NGINX Ingress Controller configures NGINX to use all referenced rate limits. When you define multiple policies, each additional policy inherits the `dryRun`, `logLevel`, and `rejectCode` parameters from the first policy referenced (`rate-limit-policy-one`, in the example above). ### RateLimit.Condition RateLimit.Condition defines a condition for a rate limit policy. For example: ```yaml condition: jwt: claim: user_details.level match: premium default: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``jwt`` | defines a JWT condition to rate limit against. | [ratelimit.condition.jwt](#ratelimitconditionjwt) | No | |``variables`` | defines a Variable condition to rate limit against. | [ratelimit.condition.variables](#ratelimitconditionvariables) | No | |``default`` | sets the rate limit in this policy to be the default if no conditions are met. In a group of policies with the same condition, only one policy can be the default. | ``bool`` | No | **Note:** Conditions (`jwt` or `variables`) are optional, but each policy can only have one. If conditions are used and a request doesn't match any of them, NGINX Ingress Controller applies the `default` policy, if one is defined. Otherwise, if no `default` is set, the request isn't rate limited. Combine the rate limit policy with condition with one or more rate limit policies. For example, you can combine multiple rate limit policies that use [RateLimit.Condition.JWT](#ratelimitconditionjwt) to apply different tiers of rate limit based on the value of a JWT claim. For a practical example of tiered rate limiting by the value of a JWT claim, see the example in the [GitHub repository](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/rate-limit-tiered-jwt-claim/README.md). ### RateLimit.Condition.JWT **Note:** This feature is only available with NGINX Plus. RateLimit.Condition.JWT defines a condition for a rate limit by JWT claim. For example, the following condition applies a rate limit policy only to requests with a JWT claim `user_details.level` with a value `premium`: ```yaml jwt: claim: user_details.level match: premium ``` The rate limit policy applies only to requests that contain a JWT with the specified claim and value. For example, the following JWT payload matches the JWT condition: ```json { "user_details": { "level": "premium" }, "sub": "client1" } ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``claim`` | Claim is the JWT claim on which to apply the rate limit. Nested claims should be separated by ".". | ``string`` | Yes | |``match`` | the value of the claim to match against. | ``string`` | Yes | ### RateLimit.Condition.Variables RateLimit.Condition.Variables defines a condition for a rate limit by NGINX variable. The following example defines a condition for a rate limit policy that applies only to requests with the request method with a value `GET`: ```yaml variables: - name: $request_method match: GET ``` **Note:** NGINX Ingress Controller currently supports only one variable at a time. |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | the name of the NGINX variable on which to apply the rate limit. | ``string`` | Yes | |``match`` | the value of the NGINX variable to match against. Values prefixed with the `~` character denote the following is a [regular expression](https://nginx.org/en/docs/http/ngx_http_map_module.html#map). | ``string`` | Yes | ## APIKey The API Key auth policy configures NGINX to authorize client requests based on the presence of a valid API Key in a header or query parameter specified in the policy. **Note:** This feature uses the NGINX [ngx_http_auth_request_module](https://nginx.org/en/docs/http/ngx_http_auth_request_module.html) and [NGINX JavaScript (NJS)](https://nginx.org/en/docs/njs/). Subrequests may not function as expected and may cause issues when the `APIKey` policy and a `WAF` policy are applied together on the same route. The policy stores API keys securely using SHA-256 hashing. When a client sends an API Key, NJS hashes it and compares it to the hashed API Key in the NGINX configuration. If the hashed keys match, the NJS subrequest issues a 204 No Content response to the `auth_request` directive, indicating successful authorization. If the client doesn't provide an API Key in the specified header or query parameter, NGINX returns a 401 Unauthorized response. If the client presents an invalid key in the expected header or query parameter, NGINX returns a 403 Forbidden response and denies access. You can use the [errorPages](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#errorpage) property on a route to change the default behavior for 401 or 403 errors. The policy requires at least one header or query parameter. The policy below configures NGINX Ingress Controller to require the API Key `password` in the header "my-header". ```yaml apiKey: suppliedIn: header: - "my-header" clientSecret: api-key-secret ``` ```yaml apiVersion: v1 kind: Secret metadata: name: api-key-secret type: nginx.org/apikey data: client1: cGFzc3dvcmQ= # password ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``suppliedIn`` | `header` or `query`. | | Yes | |``suppliedIn.header`` | An array of headers that the API Key may appear in. | ``string[]`` | No | |``suppliedIn.query`` | An array of query params that the API Key may appear in. | ``string[]`` | No | |``clientSecret`` | The name of the Kubernetes secret that stores the API Key(s). It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/apikey``, and the API Key(s) must be stored in a key: val format where each key is a unique clientID and each value is a unique base64 encoded API Key | ``string`` | Yes | **Note:** An APIKey policy must include at least one of the `suppliedIn.header` or `suppliedIn.query` parameters. You can also include both. ### APIKey merging behavior A VirtualServer or VirtualServerRoute can be associated with only one API Key policy per route or subroute. You can replace an API Key policy from a higher level with a different policy defined on a more specific route. For example, a VirtualServer can implement different API Key policies at various levels. In the following configuration, the server-wide `api-key-policy-server` applies to `/backend1` for authorization, because that route has no more specific policy. `/backend2` uses `api-key-policy-route`, defined at the route level. ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: virtual-server spec: host: virtual-server.example.com policies: - name: api-key-policy-server upstreams: - name: backend2 service: backend2-svc port: 80 - name: backend1 service: backend1-svc port: 80 routes: - path: /backend1 action: pass: backend1 - path: /backend2 action: pass: backend2 policies: - name: api-key-policy-route ``` ## BasicAuth The basic auth policy configures NGINX to authenticate client requests using the [HTTP Basic authentication scheme](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication). For example, the following policy rejects all requests that don't include a valid username and password combination in the HTTP header `Authentication`: ```yaml basicAuth: secret: htpasswd-secret realm: "My API" ``` **Note:** This feature uses the NGINX [ngx_http_auth_basic_module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html). |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of the Kubernetes secret that stores the Htpasswd configuration. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/htpasswd``, and the config must be stored in the secret under the key ``htpasswd``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. | ``string`` | Yes | |``realm`` | The realm for the basic authentication. | ``string`` | No | ### BasicAuth merging behavior A VirtualServer or VirtualServerRoute can reference multiple basic auth policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: basic-auth-policy-one - name: basic-auth-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `basic-auth-policy-one`, and ignores `basic-auth-policy-two`. ## JWT using a local Kubernetes secret **Note:** This feature is only available with NGINX Plus. The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. The following example policy rejects all requests that don't include a valid JWT in the HTTP header `token`: ```yaml jwt: secret: jwk-secret realm: "My API" token: $http_token ``` You can pass the JWT claims and JOSE headers to the upstream servers. For example: ```yaml action: proxy: upstream: webapp requestHeaders: set: - name: user value: ${jwt_claim_user} - name: alg value: ${jwt_header_alg} ``` This example uses the `requestHeaders` of [Action.Proxy](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#actionproxy) to set the values of two headers that NGINX passes to the upstream servers. The value of the `${jwt_claim_user}` variable is the `user` claim of a JWT. For other claims, use `${jwt_claim_name}`, where `name` is the name of the claim. Nested claims and claims that include a period (`.`) aren't supported. Similarly, use `${jwt_header_name}`, where `name` is the name of a header. This example uses the `alg` header. **Note:** This feature uses the NGINX Plus [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of the Kubernetes secret that stores the JWK. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/jwk``, and the JWK must be stored in the secret under the key ``jwk``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. | ``string`` | Yes | |``realm`` | The realm of the JWT. | ``string`` | Yes | |``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | ### JWT merging behavior A VirtualServer or VirtualServerRoute can reference multiple JWT policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: jwt-policy-one - name: jwt-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `jwt-policy-one`, and ignores `jwt-policy-two`. ## JWT using JWKS from a remote location **Note:** This feature is only available with NGINX Plus. The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. You can import the JWKS keys for a JWT policy from a URL, such as a remote server or an identity provider, so you don't have to copy and update them on the Ingress Controller pod. The following example policy rejects all requests that don't include a valid JWT in the HTTP header fetched from the identity provider: ```yaml jwt: realm: MyProductAPI token: $http_token jwksURI: keyCache: 1h ``` **Note:** This feature uses the NGINX Plus directive [auth_jwt_key_request](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html#auth_jwt_key_request), part of [ngx_http_auth_jwt_module](https://nginx.org/en/docs/http/ngx_http_auth_jwt_module.html). Subrequests may not function as expected and may cause issues when fetching JWKs from a remote URI (`jwksURI`) in a `JWT` policy and a `WAF` policy are applied together on the same route. |Field | Description | Type | Required | Default | | ---| ---| ---| --- | --- | |``jwksURI`` | The remote URI where NGINX Ingress Controller sends the request to retrieve the JSON Web Key set.| ``string`` | Yes | -- | |``keyCache`` | Turns on in-memory caching of JWKS (JSON Web Key Sets) obtained from the ``jwksURI`` and sets a valid time for expiration. | ``string`` | Yes | -- | |``realm`` | The realm of the JWT. | ``string`` | Yes | -- | |``token`` | The token specifies a variable that contains the JSON Web Token. By default the JWT is passed in the ``Authorization`` header as a Bearer Token. JWT may be also passed as a cookie or a part of a query string, for example: ``$cookie_auth_token``. Accepted variables are ``$http_``, ``$arg_``, ``$cookie_``. | ``string`` | No | -- | |``sniEnabled`` | Turns on SNI (Server Name Indication) for the JWT policy. Use this when the remote server requires SNI to serve the correct certificate. | ``bool`` | No | `false` | |``sniName`` | The SNI name to use when connecting to the remote server. If not set, NGINX Ingress Controller uses the hostname from the ``jwksURI``. | ``string`` | No | -- | |``sslVerify`` | Turns on verification of the JWKS server SSL certificate. | ``bool`` | No | `false` | |``sslVerifyDepth`` | Sets the verification depth in the JWKS server certificates chain. | ``int`` | No | `1` | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate for JWKS server verification. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. | ``string`` | No | -- | **Note:** NGINX Ingress Controller turns on content caching by default for each JWT policy, with a default time of 12 hours. This improves resiliency by letting NGINX Ingress Controller retrieve the JWKS (JSON Web Key Set) from the cache even after it expires. ### JWT using JWKS merging behavior This behavior is similar to using a local Kubernetes secret. A VirtualServer or VirtualServerRoute can reference multiple JWT policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: jwt-policy-one - name: jwt-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `jwt-policy-one`, and ignores `jwt-policy-two`. ## IngressMTLS The IngressMTLS policy configures client certificate verification. For example, the following policy verifies a client certificate using the CA certificate specified in `ingress-mtls-secret`: ```yaml ingressMTLS: clientCertSecret: ingress-mtls-secret verifyClient: "on" verifyDepth: 1 ``` Below is an example of `ingress-mtls-secret` using the secret type `nginx.org/ca`: ```yaml kind: Secret metadata: name: ingress-mtls-secret apiVersion: v1 type: nginx.org/ca data: ca.crt: ``` A VirtualServer that references an IngressMTLS policy must: - Turn on [TLS termination](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualservertls). - Reference the policy in the VirtualServer [`spec`](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserver-specification). You can't reference an IngressMTLS policy in a [`route`](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroute) or in a VirtualServerRoute [`subroute`](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroutesubroute). A Kubernetes Ingress that references an IngressMTLS policy must: - Turn on [TLS termination](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#auth-and-ssltls). - Reference the policy on the Ingress. For [mergeable Ingresses](/nic/configuration/ingress-resources/custom-annotations.md#custom-annotations-with-mergeable-ingress-resources), reference the policy on the master Ingress only. You can't reference an IngressMTLS policy on a minion Ingress. If a resource doesn't meet these conditions, NGINX sends the `500` status code to clients. You can pass the client certificate details, including the certificate, to the upstream servers. For example: ```yaml action: proxy: upstream: webapp requestHeaders: set: - name: client-cert-subj-dn value: ${ssl_client_s_dn} # subject DN - name: client-cert value: ${ssl_client_escaped_cert} # client certificate in the PEM format (urlencoded) ``` This example uses the `requestHeaders` of [Action.Proxy](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#actionproxy) to set the values of the two headers that NGINX passes to the upstream servers. See the [list of embedded variables](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#variables) that `ngx_http_ssl_module` supports, which you can use to pass the client certificate details. **Note:** This feature uses the NGINX [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html). ### Configure a certificate revocation list The IngressMTLS policy supports configuring a certificate revocation list (CRL) for your policy, in one of two ways. **Note:** You can use only one of these configuration options at a time. 1. Add the `ca.crl` field to the `nginx.org/ca` secret type, which accepts a base64 encoded certificate revocation list. Example: ```yaml kind: Secret metadata: name: ingress-mtls-secret apiVersion: v1 type: nginx.org/ca data: ca.crt: ca.crl: ``` 2. Add the `crlFileName` field to your IngressMTLS policy spec with the name of the CRL file. **Note:** Use this configuration option only when your CRL is larger than 1 MiB. Otherwise, use the `nginx.org/ca` secret type to manage your CRL. Example: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: ingress-mtls-policy spec: ingressMTLS: clientCertSecret: ingress-mtls-secret crlFileName: webapp.crl verifyClient: "on" verifyDepth: 1 ``` **Note:** When you configure a CRL with the `ingressMTLS.crlFileName` field, keep this additional context in mind: - NGINX Ingress Controller expects the CRL, in this case `webapp.crl`, to be in `/etc/nginx/secrets`. Add a volume mount to the NGINX Ingress Controller deployment to add your CRL to `/etc/nginx/secrets`. - When you update the content of your CRL, for example after you revoke a new certificate, NGINX needs to reload to pick up the latest changes. Depending on your environment, this may require you to update the name of your CRL and apply this update to your `ingress-mtls.yaml` policy so NGINX picks up the latest CRL. See the Kubernetes documentation on [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) to find the best implementation for your environment. |Field | Description | Type | Required | | ---| ---| ---| --- | |``clientCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. | ``string`` | Yes | |``verifyClient`` | Verification for the client. Possible values are ``"on"``, ``"off"``, ``"optional"``, ``"optional_no_ca"``. The default is ``"on"``. | ``string`` | No | |``verifyDepth`` | Sets the verification depth in the client certificates chain. The default is ``1``. | ``int`` | No | |``crlFileName`` | The file name of the Certificate Revocation List. NGINX Ingress Controller looks for this file in `/etc/nginx/secrets`. | ``string`` | No | ### IngressMTLS merging behavior A VirtualServer or an Ingress can reference only a single IngressMTLS policy, and NGINX Ingress Controller ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: ingress-mtls-policy-one - name: ingress-mtls-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `ingress-mtls-policy-one`, and ignores `ingress-mtls-policy-two`. ## EgressMTLS The EgressMTLS policy configures upstream authentication and certificate verification. For example, the following policy uses `egress-mtls-secret` to authenticate with the upstream application and `egress-trusted-ca-secret` to verify the certificate of the application: ```yaml egressMTLS: tlsSecret: egress-mtls-secret trustedCertSecret: egress-trusted-ca-secret verifyServer: on verifyDepth: 2 ``` **Note:** This feature uses the NGINX [ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html). |Field | Description | Type | Required | | ---| ---| ---| --- | |``tlsSecret`` | The name of the Kubernetes secret that stores the TLS certificate and key. It must be in the same namespace as the Policy resource. The secret must be of the type ``kubernetes.io/tls``, the certificate must be stored in the secret under the key ``tls.crt``, and the key must be stored under the key ``tls.key``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. | ``string`` | No | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. | ``string`` | No | |``verifyServer`` | Turns on verification of the upstream HTTPS server certificate. | ``bool`` | No | |``verifyDepth`` | Sets the verification depth in the proxied HTTPS server certificates chain. The default is ``1``. | ``int`` | No | |``sessionReuse`` | Turns on reuse of SSL sessions to the upstreams. The default is ``true``. | ``bool`` | No | |``serverName`` | Turns on passing of the server name through the ``Server Name Indication`` extension. | ``bool`` | No | |``sslName`` | Lets you override the server name used to verify the certificate of the upstream HTTPS server. | ``string`` | No | |``ciphers`` | Specifies the enabled ciphers for requests to an upstream HTTPS server. The default is ``DEFAULT``. | ``string`` | No | |``protocols`` | Specifies the protocols for requests to an upstream HTTPS server. The default is ``TLSv1 TLSv1.1 TLSv1.2``. | ``string`` | No | > Note: NGINX Ingress Controller doesn't validate the value of ``ciphers`` and ``protocols``. As a result, NGINX can fail to reload the configuration. To make sure the configuration for a VirtualServer or VirtualServerRoute that references the policy applied successfully, check its [status](/nic/configuration/global-configuration/reporting-resources-status.md#virtualserver-and-virtualserverroute-resources). Validation for these fields is planned for a future release. | ### EgressMTLS merging behavior A VirtualServer or VirtualServerRoute can reference multiple EgressMTLS policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: egress-mtls-policy-one - name: egress-mtls-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `egress-mtls-policy-one`, and ignores `egress-mtls-policy-two`. ## ExternalAuth **Note:** Subrequests may not function as expected and may cause issues when the `ExternalAuth` policy and a `WAF` policy are applied together on the same route. The ExternalAuth policy configures NGINX to authenticate client requests using an external authentication server. You can use this policy with services such as [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/) or any custom authentication service that supports the `auth_request` pattern. When a client sends a request, NGINX makes an internal subrequest to the external authentication service. If the service returns a `2xx` response, NGINX forwards the original request to the upstream. If it returns `401` or `403`, NGINX denies access. If you configure `authSigninURI`, NGINX redirects unauthenticated clients to a sign-in page. For example, the following policy configures external authentication using an HTTP Basic Auth backend service: ```yaml externalAuth: authURI: "/auth" authServiceName: "default/basic-auth-svc" ``` The following policy uses OAuth2 Proxy with a sign-in redirect: ```yaml externalAuth: authURI: "/oauth2/auth" authSigninURI: "/oauth2/signin" authServiceName: "default/oauth2-proxy-svc" sslEnabled: true sslVerify: true sslVerifyDepth: 2 sniName: "external-auth-tls" trustedCertSecret: "external-auth-ca-secret" ``` An example ExternalAuth policy for VirtualServer resources is available in the GitHub repository for [basic auth](https://github.com/nginx/kubernetes-ingress/blob/v/examples/custom-resources/external-auth) and [OAuth2 with basic auth](https://github.com/nginx/kubernetes-ingress/blob/v/examples/custom-resources/external-auth-oauth2). Examples for Ingress resources are also available for [basic auth](https://github.com/nginx/kubernetes-ingress/blob/v/examples/ingress-resources/external-auth) and [OAuth2 with basic auth using mergeable Ingresses](https://github.com/nginx/kubernetes-ingress/blob/v/examples/ingress-resources/external-auth-mergeable). |Field | Description | Type | Required | | ---| ---| ---| --- | |``authURI`` | The URI of the external authentication server. NGINX sends an internal subrequest to this URI to verify the client. Must start with ``/``. For example, ``/auth`` or ``/oauth2/auth``. | ``string`` | Yes | |``authServiceName`` | The name of the Kubernetes service for the external authentication server. Can include an optional namespace prefix in the format ``/``. For example, ``basic-auth-svc`` or ``auth-namespace/auth-service``. If no namespace is specified, the namespace of the Policy resource is used. | ``string`` | Yes | |``authServicePorts`` | The ports of the Kubernetes service to which authentication requests are sent. If not specified, the first port from the service definition is used. | ``[]int`` | No | |``authSigninURI`` | The URI to redirect unauthenticated clients to for sign-in. Used when the external authentication server requires redirection, such as with OAuth2 Proxy. Must start with ``/``. For example, ``/oauth2/signin``. | ``string`` | No | |``authSnippets`` | Custom NGINX configuration snippets to add to the external authentication location block. For example, you can add extra headers or parameters for the ``auth_request`` module. The content must be valid NGINX configuration. Requires the [``-enable-snippets``](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-snippets) command-line argument. | ``string`` | No | |``authSigninRedirectBasePath`` | The base path for the NGINX location block that handles sign-in redirect requests from the external authentication server. For example, OAuth2 Proxy expects ``/oauth2``. Defaults to ``/oauth2`` if not specified. | ``string`` | No | |``sslEnabled`` | Turns on HTTPS when proxying requests to the external authentication server. The default is ``false``. | ``bool`` | No | |``sslVerify`` | Turns on verification of the external authentication server's SSL certificate. The default is ``false``. | ``bool`` | No | |``sslVerifyDepth`` | Sets the verification depth in the external authentication server certificates chain. The default is ``1``. | ``int`` | No | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate for external authentication server certificate verification. Can include an optional namespace prefix as ``/``. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored under the key ``ca.crt``. | ``string`` | No | |``sniName`` | The server name used for SNI and certificate verification when connecting to the external authentication server over TLS. If not specified, defaults to ``..svc`` derived from ``authServiceName``. | ``string`` | No | ### ExternalAuth merging behavior A VirtualServer, VirtualServerRoute, Ingress, or mergeable Ingress can reference only one ExternalAuth policy per route, and NGINX Ingress Controller ignores every subsequent reference. This means you can't combine different types of external authentication on the same route. For example, you can't apply both an OAuth2 policy and a basic auth policy to the same route. ```yaml policies: - name: external-auth-policy-one - name: external-auth-policy-two ``` NGINX Ingress Controller uses the configuration from the first policy reference, `external-auth-policy-one`, and ignores `external-auth-policy-two`. To use different authentication methods on different routes, apply each ExternalAuth policy to its own route. ### OAuth2 sign-in redirect location When you configure `authSigninURI`, NGINX Ingress Controller generates an internal location block to handle sign-in redirects, based on `authSigninRedirectBasePath`, which defaults to `/oauth2`. Because NGINX location blocks are defined at the server level, only one sign-in redirect location can exist per host. If multiple routes on the same VirtualServer, VirtualServerRoute, Ingress, or mergeable Ingress host reference ExternalAuth policies with `authSigninURI`, NGINX Ingress Controller uses the sign-in redirect configuration from the first policy it processes. All routes that use `authSigninURI` share that single redirect location. This means all routes on the same host that require OAuth2 sign-in must use the same OAuth2 Proxy backend for the redirect flow. If you need different OAuth2 providers for different routes, use separate hosts. ## OIDC **Note:** This feature is turned off by default. To turn it on, set the [enable-oidc](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-oidc) command-line argument of NGINX Ingress Controller. **Note:** Subrequests may not function as expected and may cause issues when the `OIDC` policy and a `WAF` policy are applied together on the same route. The OIDC policy configures NGINX Plus as a relying party for OpenID Connect authentication. For example, the following policy uses the client ID `nginx-plus` and the client secret `oidc-secret` to authenticate with the OpenID Connect provider `https://idp.example.com`: ```yaml spec: oidc: clientID: nginx-plus clientSecret: oidc-secret authEndpoint: https://idp.example.com/openid-connect/auth tokenEndpoint: https://idp.example.com/openid-connect/token jwksURI: https://idp.example.com/openid-connect/certs endSessionEndpoint: https://idp.example.com/openid-connect/logout postLogoutRedirectURI: / accessTokenEnable: true pkceEnable: false ``` NGINX Plus passes the ID of an authenticated user to the backend in the HTTP header `username`. **Note:** This feature uses the [reference implementation](https://github.com/nginxinc/nginx-openid-connect/) of NGINX Plus as a relying party for OpenID Connect authentication. ### Prerequisites To use OIDC, turn on [zone synchronization](/nginx/admin-guide/high-availability/zone_sync.md). If you don't set up zone synchronization, NGINX Plus fails to reload. You also need to configure a resolver, which NGINX Plus uses to resolve the IDP authorization endpoint. You can find an example configuration [in the GitHub repository](https://github.com/nginx/kubernetes-ingress/blob/v/examples/custom-resources/oidc#step-7---configure-nginx-plus-zone-synchronization-and-resolver). **Note:** The configuration in the example doesn't turn on TLS, so synchronization between replicas happens in clear text. This can expose tokens. ### Limitations The OIDC policy defines a few internal locations that you can't customize: `/_jwks_uri`, `/_token`, `/_refresh`, `/_id_token_validation`, `/logout`. In addition, `/_codexch` is the default value for the redirect URI, and `/_logout` is the default value for the post logout redirect URI. You can customize both. Specifying one of these locations as a route in the VirtualServer or VirtualServerRoute causes a collision, and NGINX Plus fails to reload. |Field | Description | Type | Required | | ---| ---| ---| --- | |``clientID`` | The client ID provided by your OpenID Connect provider. | ``string`` | Yes | |``clientSecret`` | The name of the Kubernetes secret that stores the client secret provided by your OpenID Connect provider. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/oidc``, and the secret stored under the key ``client-secret``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. If you enable PKCE, don't configure this field. | ``string`` | Yes | |``authEndpoint`` | URL for the authorization endpoint provided by your OpenID Connect provider. | ``string`` | Yes | |``authExtraArgs`` | A list of extra URL arguments to pass to the authorization endpoint provided by your OpenID Connect provider. Arguments must be URL encoded, multiple arguments may be included in the list, for example ``[ arg1=value1, arg2=value2 ]`` | ``string[]`` | No | |``tokenEndpoint`` | URL for the token endpoint provided by your OpenID Connect provider. | ``string`` | Yes | |``endSessionEndpoint`` | URL provided by your OpenID Connect provider to request the end user be logged out. | ``string`` | No | |``jwksURI`` | URL for the JSON Web Key Set (JWK) document provided by your OpenID Connect provider. | ``string`` | Yes | |``scope`` | List of OpenID Connect scopes. The scope ``openid`` always needs to be present and others can be added concatenating them with a ``+`` sign, for example ``openid+profile+email``, ``openid+email+userDefinedScope``. The default is ``openid``. | ``string`` | No | |``redirectURI`` | Lets you override the default redirect URI. The default is ``/_codexch``. | ``string`` | No | |``postLogoutRedirectURI`` | URI to redirect to after the logout has been performed. Requires ``endSessionEndpoint``. The default is ``/_logout``. | ``string`` | No | |``zoneSyncLeeway`` | Specifies the maximum timeout in milliseconds for synchronizing ID/access tokens and shared values between Ingress Controller pods. The default is ``200``. | ``int`` | No | |``accessTokenEnable`` | Option of whether Bearer token is used to authorize NGINX to access protected backend. | ``boolean`` | No | |``pkceEnable`` | Turns on Proof Key for Code Exchange. The OpenID client needs to be in public mode. `clientSecret` is not used in this mode. | ``boolean`` | No | |``sslVerify`` | Use this option to turn on TLS verification when calls are made to the IDP endpoints. | ``boolean`` | No | |``verifyDepth`` | Sets the verification depth in the proxied HTTPS server certificates chain. The default is ``1``. | ``int`` | No | |``trustedCertSecret`` | The name of the Kubernetes secret that stores the CA certificate. It must be in the same namespace as the Policy resource. The secret must be of the type ``nginx.org/ca``, and the certificate must be stored in the secret under the key ``ca.crt``. Otherwise, NGINX Ingress Controller rejects the secret as invalid. | ``string`` | No | **Note:** Only one OIDC policy can be referenced in a VirtualServer and its VirtualServerRoutes. However, you can still apply the same policy to different routes in the VirtualServer and VirtualServerRoutes. ### OIDC merging behavior A VirtualServer or VirtualServerRoute can reference only a single OIDC policy, and NGINX Ingress Controller ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: oidc-policy-one - name: oidc-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `oidc-policy-one`, and ignores `oidc-policy-two`. ## OIDCNative **Note:** This feature is only available with NGINX Plus and requires the [enable-oidc](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-enable-oidc) command-line argument. The OIDCNative policy configures NGINX Plus as a relying party for OpenID Connect authentication using the built-in `ngx_http_oidc_module`. Unlike the NJS-based [`oidc`](#oidc) policy, the native implementation handles the entire OIDC flow within the NGINX core, including token exchange, session management, and front-channel logout. For example, the following policy authenticates users against a Keycloak identity provider: ```yaml spec: oidcNative: issuer: https://idp.example.com/realms/master clientID: nginx-plus clientSecret: oidc-secret scope: openid profile email logoutURI: /logout postLogoutRedirectURI: /_logout ``` **Note:** This feature uses the NGINX [ngx_http_oidc_module](https://nginx.org/en/docs/http/ngx_http_oidc_module.html). ### Prerequisites Configure a resolver so NGINX Plus can resolve the identity provider's hostname for discovery and token exchange. Add the following to your ConfigMap: ```yaml data: resolver-addresses: "kube-dns.kube-system.svc.cluster.local" ``` When you use the [zone synchronization](/nic/configuration/global-configuration/configmap-resource.md#zone-sync) feature, NGINX Ingress Controller synchronizes OIDC session data across all replicas. Put the ConfigMap in its final state, with or without zone sync turned on, before you create the OIDCNative policy and its referencing VirtualServer or Ingress. NGINX declares the native module's session zone with sync only when zone sync is turned on, and NGINX can't change the sync flag of an existing shared memory zone across a reload. ### Differences from the NJS-based OIDC policy | Aspect | OIDC (NJS) | OIDCNative (native module) | | --- | --- | --- | | Implementation| NGINX JavaScript reference implementation | Built-in `ngx_http_oidc_module` | | Discovery | Manual endpoint configuration (`authEndpoint`, `tokenEndpoint`, `jwksURI`) | Automatic through OpenID Connect Discovery (`issuer` or `configURL`) | | Ingress Support | No | Yes, through `nginx.com/policies` annotation | | Front-channel logout | Through NJS handler | Through `frontChannelLogoutURI` directive | | PKCE | `pkceEnable` boolean | `pkce` enum (on/off), auto-detected from provider metadata by default | | Multiple providers per host | One per VirtualServer | Multiple (unique provider per policy and resource combination) | **Note:** `oidc` and `oidcNative` are mutually exclusive. A Policy resource must define only one of them. If you set both, NGINX Ingress Controller marks the Policy as Invalid. On a VirtualServer, if you apply both an `oidc` policy and an `oidcNative` policy to the same route, that route gets a Warning status and returns a static 500 response. | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `issuer` | The Issuer Identifier URL of the OpenID Provider. Must use the `https` scheme and exactly match the value of `issuer` in the OpenID Provider metadata. | `string` | Yes | -- | | `clientID` | The client ID provided by your OpenID Connect provider. | `string` | Yes | -- | | `clientSecret` | The name of the Kubernetes secret that stores the client secret. Must be of type `nginx.org/oidc` with the secret stored under the key `client-secret` and must be in the same namespace as the Policy resource. Not required when PKCE is enabled with a public client. | `string` | No | -- | | `configURL` | The URL of the OpenID Provider Configuration Information (discovery endpoint). Must include a path and use the `http` or `https` scheme. If not set, defaults to `/.well-known/openid-configuration`. | `string` | No | `/.well-known/openid-configuration` | | `scope` | Space-separated list of OpenID Connect scopes. Must contain `openid`. Example: `"openid profile email"`. | `string` | No | `openid` | | `redirectURI` | Overrides the default redirect URI path used for the authorization callback. | `string` | No | `/oidc_callback_` | | `cookieName` | Sets the name of the session cookie. Must contain only letters, digits, and underscores. | `string` | No | `NGX_OIDC_` | | `extraAuthArgs` | Additional query arguments appended to the authorization request URL. Example: `"display=page&prompt=login"`. | `string` | No | -- | | `pkce` | Explicitly turns PKCE on or off. By default, NGINX Ingress Controller turns on PKCE automatically based on OpenID Provider metadata. | `string (on/off)` | No | -- | | `logoutURI` | URI path for initiating session logout. Session logout is unavailable until this field is set. Use a path the attached route can serve, for example `/logout` for a policy on the `/` route, or `/tea/logout` for a policy on the `/tea` route. A path outside the attached route isn't matched by that route's location, so logout doesn't trigger. | `string` | No | -- | | `postLogoutRedirectURI` | Path the user is returned to after logout completes. Must be a path on the same host, absolute URLs aren't supported. NGINX Ingress Controller generates an unauthenticated location at this path serving a plain-text confirmation, shared if multiple providers use the same path. | `string` | No | -- | | `frontChannelLogoutURI` | URI path for OIDC front-channel logout. When set, the identity provider calls this URI in a hidden iframe during global logout, letting NGINX end the local session. | `string` | No | -- | | `logoutTokenHint` | Adds the `id_token_hint` argument to the provider's logout endpoint when redirecting the user during logout. Required by some providers. | `bool` | No | `false` | | `sessionTimeout` | Duration after which the session expires unless refreshed. Example: `"8h"`, `"30m"`. | `string` | No | `8h` | | `userInfoEnable` | Turns on downloading of the UserInfo data and makes UserInfo claims available through `$oidc_claim_` variables. | `bool` | No | `false` | | `trustedCertSecret` | The name of the Kubernetes secret that stores the CA certificate for verifying the provider's TLS certificate. Must be of type `nginx.org/ca` with the certificate stored under key `ca.crt`. | `string` | No | -- | | `sslVerify` | Turns on verification of the OpenID Provider's TLS certificate. Set to false to skip verification (dev/test only). | `bool` | No | `true` | | `sslName` | Overrides the TLS SNI name and Host header used when connecting to the OpenID Provider. Must be a valid DNS name and can't include a port. When unset, the hostname of the endpoint being called is used, taken from the provider's discovery metadata. | `string` | No | -- | | `sslVerifyDepth` | Verification depth in the OpenID Provider TLS certificate chain. | `int` | No | `1` | | `proxyBufferSize` | Buffer size used when proxying requests to the OpenID Provider. Applies to proxy_buffer_size and each buffer in proxy_buffers. | `string` | No | `32k` | ### OIDCNative on Ingress When using OIDCNative with Ingress resources, reference the policy through the `nginx.com/policies` annotation: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webapp annotations: nginx.com/policies: "oidc-native-policy" spec: ingressClassName: nginx tls: - hosts: - webapp.example.com secretName: tls-secret rules: - host: webapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: webapp-svc port: number: 80 ``` For mergeable Ingress, you can apply the policy at the master level (server-wide) or on individual minions (location-level): ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webapp-master annotations: nginx.org/mergeable-ingress-type: "master" nginx.com/policies: "oidc-native-policy" spec: ingressClassName: nginx tls: - hosts: - webapp.example.com secretName: tls-secret rules: - host: webapp.example.com ``` **Note:** On Ingress, reference OIDCNative policies through `nginx.com/policies`, not `nginx.org/policies`. Ingress resources don't support the NJS-based `oidc` policy. ### OIDCNative merging behavior A VirtualServer, VirtualServerRoute, or Ingress can reference only a single OIDCNative policy per context, and NGINX Ingress Controller ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: oidc-policy-one - name: oidc-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `oidc-policy-one`, and ignores `oidc-policy-two`. Multiple OIDCNative policies can coexist on the same host when applied to different routes (VirtualServer) or different mergeable Ingress minions. Each generates a unique `oidc_provider` block with its own session store. If you override those paths and two policies on the same host claim the same one, for example by setting `redirectURI` to the same value, NGINX Ingress Controller rejects the conflicting policy and that route or minion returns a 500 response. `postLogoutRedirectURI` is the exception: providers may share it, and only one location is generated. ## Cache The cache policy configures proxy caching, which improves performance by storing and serving cached responses to clients instead of proxying every request to upstream servers. **Note:** Subrequests may not function as expected and may cause issues when `cacheBackgroundUpdate` in a `Cache` policy and a `WAF` policy are applied together on the same route. For example, the following policy creates a cache zone named "mycache" with 10 MB of memory allocated, and caches all GET response codes for 30 seconds: ```yaml cache: cacheZoneName: "mycache" cacheZoneSize: "10m" allowedCodes: ["any"] allowedMethods: ["GET"] time: "30s" ``` Here's an example with more specific configuration: ```yaml cache: cacheZoneName: "mycache" cacheZoneSize: "100m" allowedCodes: [200, 301, 302] allowedMethods: ["GET", "POST"] time: "5m" levels: "1:2" overrideUpstreamCache: true inactive: "60m" useTempPath: false maxSize: "10g" minFree: "1g" manager: files: 100 sleep: "50ms" threshold: "200ms" cacheKey: "$scheme$host$request_uri" cacheUseStale: [ "error", "timeout", "updating", "http_500" ] cacheRevalidate: true cacheBackgroundUpdate: true cacheMinUses: 1 lock: enable: true timeout: "5s" age: "30s" conditions: noCache: [ "$cookie_nocache", "$arg_nocache" ] bypass: [ "$http_authorization" ] ``` **Note:** This feature uses the NGINX [ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path) `proxy_cache_path` and related directives. |Field | Description | Type | Required | | --- | ---| ---| --- | |``cacheZoneName`` | CacheZoneName defines the name of the cache zone. Must start with a lowercase letter, followed by alphanumeric characters or underscores, and end with an alphanumeric character. Single lowercase letters are also allowed. Examples: "cache", "my_cache", "cache1". | ``string`` | Yes | |``cacheZoneSize`` | CacheZoneSize defines the size of the cache zone. Must be a number followed by a size unit: 'k' for kilobytes, 'm' for megabytes, or 'g' for gigabytes. Examples: "10m", "1g", "512k". | ``string`` | Yes | |``allowedCodes`` | AllowedCodes defines which HTTP response codes should be cached. Accepts either: - The string "any" to cache all response codes (must be the only element) - A list of HTTP status codes as integers (100-599) Examples: ["any"], [200, 301, 404], [200]. Invalid: ["any", 200] (cannot mix "any" with specific codes). | ``[]IntOrString`` | No | |``time`` | The default cache time for responses. Required when allowedCodes is specified. Must be a number followed by a time unit: 's' for seconds, 'm' for minutes, 'h' for hours, 'd' for days. Examples: "30s", "5m", "1h", "2d". | ``string`` | No | |``allowedMethods`` | AllowedMethods defines which HTTP methods should be cached. Only "GET", "HEAD", and "POST" are supported by the NGINX `proxy_cache_methods` directive. GET and HEAD are always cached by default even if not specified. Maximum of 3 items allowed. Examples: ["GET"], ["GET", "HEAD", "POST"]. Invalid methods: PUT, DELETE, PATCH, and so on. | ``[]string`` | No | |``levels`` | Levels defines the cache directory hierarchy levels for storing cached files. Must be in format "X:Y" or "X:Y:Z" where X, Y, Z are either 1 or 2. This controls the number of subdirectory levels and their name lengths. Examples: "1:2", "2:2", "1:2:2". Invalid: "3:1", "1:3", "1:2:3". | ``string`` | No | |``overrideUpstreamCache`` | OverrideUpstreamCache controls whether to override upstream cache headers (using the `proxy_ignore_headers` directive). When true, NGINX ignores cache-related headers from upstream servers like Cache-Control, Expires, and so on. Default: false. | ``bool`` | No | |``cachePurgeAllow`` | CachePurgeAllow defines IP addresses or CIDR blocks allowed to purge cache. This feature is only available in NGINX Plus. Examples: ["192.168.1.100", "10.0.0.0/8", "::1"]. | ``[]string`` | No | |``cacheKey`` | CacheKey defines a key for caching (`proxy_cache_key`). By default, "$scheme$proxy_host$uri". Must not contain command execution patterns: $(, `, ;, &&, || | ``string`` | No | |``cacheUseStale`` | CacheUseStale determines in which cases a stale cached response can be used (`proxy_cache_use_stale`). Valid parameters: error, timeout, invalid_header, updating, http_500, http_502, http_503, http_504, http_403, http_404, http_429, off. | ``[]string`` | No | |``cacheRevalidate`` | CacheRevalidate turns on revalidation of expired cache items using conditional requests (`proxy_cache_revalidate`). Uses "If-Modified-Since" and "If-None-Match" header fields. | ``bool`` | No | |``cacheBackgroundUpdate`` | CacheBackgroundUpdate lets NGINX start a background subrequest to update an expired cache item (`proxy_cache_background_update`). NGINX returns a stale cached response to the client while it updates the cache. | ``bool`` | No | |``cacheMinUses`` | CacheMinUses sets the number of requests after which NGINX Ingress Controller caches the response (`proxy_cache_min_uses`). | ``integer`` | No | |``inactive`` | Inactive sets the time after which cached data that are not accessed get removed from the cache (inactive parameter). By default, inactive is set to 10 minutes. | ``string`` | No | |``maxSize`` | MaxSize sets the maximum cache size (max_size parameter). When the size is exceeded, the cache manager removes the least recently used data. | ``string`` | No | |``minFree`` | MinFree sets the minimum amount of free space required on the file system with cache (min_free parameter). When there is not enough free space, the cache manager removes the least recently used data. | ``string`` | No | |``useTempPath`` | UseTempPath controls whether temporary files and the cache are put on different file systems (use_temp_path parameter). If set to false, NGINX puts temporary files directly in the cache directory (use_temp_path=off). Default: false (use_temp_path=off, which puts temp files directly in the cache directory for better performance). | ``bool`` | No | |``manager`` | Manager configures the cache manager process parameters (manager_files, manager_sleep, manager_threshold). | ``object`` | No | |``manager.files`` | Files sets the maximum number of files that the cache manager deletes in one iteration. During one iteration, the cache manager deletes no more than manager_files items (by default, 100). | ``integer`` | No | |``manager.sleep`` | Sleep sets the pause between cache manager iterations. Between iterations, a pause configured by manager_sleep (by default, 50 milliseconds) is made. | ``string`` | No | |``manager.threshold`` | Threshold sets the maximum duration of one cache manager iteration. The duration of one iteration is limited by manager_threshold (by default, 200 milliseconds). | ``string`` | No | |``lock`` | Lock configures cache locking to prevent multiple identical requests from populating the same cache element simultaneously. | ``object`` | No | |``lock.enable`` | Enable sets whether cache locking is turned on (`proxy_cache_lock`). When on, only one request at a time can populate a new cache element according to the `proxy_cache_key`. | ``bool`` | No | |``lock.timeout`` | Timeout sets a timeout for `proxy_cache_lock`. When the time expires, NGINX passes the request to the proxied server, but it doesn't cache the response. | ``string`` | No | |``lock.age`` | Age sets the maximum time a cache lock can be held (`proxy_cache_lock_age`). If the last request passed to the proxied server for populating a new cache element hasn't completed within the specified time, NGINX may pass one more request. | ``string`` | No | |``conditions`` | Conditions defines when responses should not be cached or taken from cache. | ``object`` | No | |``conditions.noCache`` | NoCache defines conditions under which the response won't be saved to a cache (`proxy_no_cache`). If at least one value of the string parameters isn't empty and isn't equal to "0", NGINX doesn't save the response. | ``[]string`` | No | |``conditions.bypass`` | Bypass defines conditions under which the response won't be taken from a cache (`proxy_cache_bypass`). If at least one value of the string parameters isn't empty and isn't equal to "0", NGINX doesn't take the response from the cache. | ``[]string`` | No | ### Cache merging behavior A VirtualServer or VirtualServerRoute can reference multiple cache policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. ## CORS The CORS policy configures Cross-Origin Resource Sharing (CORS) headers. **Note:** This feature uses the NGINX `add_header` directive. Below is an example of a CORS policy configuring all the available options: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: cors-policy spec: cors: allowOrigin: - "https://test.example.com" - "https://app.example.com" - "https://admin.example.com" allowMethods: - "GET" - "POST" - "PUT" allowHeaders: - "Content-Type" - "Authorization" - "X-Requested-With" - "X-API-Key" allowCredentials: true exposeHeaders: - "X-Total-Count" - "X-Page-Size" - "X-RateLimit-Remaining" - "X-RateLimit-Reset" maxAge: 3600 ``` |Field | Description | Type | Required | | --- | ---| ---| --- | |``allowOrigin`` | AllowOrigin defines the origins that are allowed to make cross-origin requests. Can be exact domains, single wildcards, or `*` for all origins. Examples: ["https://example.com", "https://*.mydomain.com", "*"] Security: When allowCredentials is true, wildcard "*" is not allowed. The server must specify explicit origins for credentialed requests. |``array[string]`` | Yes | |``allowMethods`` | AllowMethods defines the HTTP methods that are allowed for cross-origin requests. | ``array[string]`` | No | |``allowHeaders`` | AllowHeaders defines the headers that are allowed in cross-origin requests. Common safe headers: ["Accept", "Accept-Language", "Content-Language", "Content-Type"] Custom headers: ["Authorization", "X-Requested-With", "X-Custom-Header"] | ``array[string]`` | No | |``allowCredentials`` | AllowCredentials indicates whether the response to the request can be exposed when the credentials flag is true. When used as part of a response to a preflight request, this indicates whether the actual request can be made using credentials. | ``boolean`` | No | |``exposeHeaders`` | ExposeHeaders defines the headers that browsers are allowed to access. Use this field to expose additional custom headers to the browser. Example: ["X-Total-Count", "X-Page-Size", "X-RateLimit-Remaining"] Note: Set-Cookie headers cannot be exposed through CORS per official MDN specification. | ``array[string]`` | No | |``maxAge`` | MaxAge defines how long (in seconds) the results of a preflight request can be cached. Default: 86400 (24 hours). | ``integer`` | No | **Note:** If CORS is currently configured in deployments using `snippets` or `responseHeaders.add`, migrate those settings to the CORS policy and remove the duplicate configuration. ### CORS merging behavior A VirtualServer or VirtualServerRoute can reference multiple CORS policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. ## WAF **Note:** This feature uses the NGINX Plus [F5 WAF for NGINX module](/waf/). **Note:** Policies that rely on NGINX subrequests (such as `ExternalAuth`, `APIKey`, `JWT` with remote JWKS fetching, `OIDC`, or `Cache` with `cacheBackgroundUpdate`) and a `WAF` policy may not function as expected and may cause issues when applied together on the same route. The WAF policy configures NGINX Plus to secure client requests using F5 WAF for NGINX policies. For example, the following policy turns on the referenced APPolicy. You can configure multiple APLogConfs with log destinations: ```yaml waf: enable: true apPolicy: "default/dataguard-alarm" securityLogs: - enable: true apLogConf: "default/logconf" logDest: "syslog:server=syslog-svc.default:514" - enable: true apLogConf: "default/logconf" logDest: "syslog:server=syslog-svc-secondary.default:514" ``` **Note:** The field `waf.securityLog` is deprecated and will be removed in a future release. NGINX Ingress Controller ignores it if `waf.securityLogs` is populated. |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Turns on F5 WAF for NGINX. | ``bool`` | Yes | |``apPolicy`` | The [F5 WAF for NGINX policy](/nic/integrations/app-protect-waf/configuration.md#waf-policies) of the WAF. References an APPolicy CR by `[/]`. When you start the Ingress Controller with `-plm-storage-url`, PLM must have compiled the referenced APPolicy (status.bundle.state == ready). Mutually exclusive with `apBundle`. | `string` | No | |``apBundle`` | The [F5 WAF for NGINX policy bundle](/nic/integrations/app-protect-waf/configuration.md#waf-bundles). Mutually exclusive with ``apPolicy`` and ``apBundleSource``. | ``string`` | No | |``apBundleSource`` | [Remote source](/nic/integrations/app-protect-waf-v5/bundle-sources.md) for fetching the WAF policy bundle. Mutually exclusive with ``apBundle`` and ``apPolicy``. | [waf.apBundleSource](#wafapbundlesource) | No | |``securityLog.enable`` | **Deprecated:** Turns on the security log. | ``bool`` | No | |``securityLog.apLogConf`` | **Deprecated:** The [F5 WAF for NGINX log conf](/nic/integrations/app-protect-waf/configuration.md#waf-logs) resource. Accepts an optional namespace. Only works with ``apPolicy``. | ``string`` | No | |``securityLog.apLogBundle`` | **Deprecated:** The [F5 WAF for NGINX log bundle](/nic/integrations/app-protect-waf/configuration.md#waf-bundles) resource. Only works with ``apBundle``. | ``string`` | No | |``securityLog.logDest`` | **Deprecated:** The log destination for the security log. Only accepted variables are ``syslog:server=; localhost; :``, ``stderr``, ````. | ``string`` | No | |``securityLogs`` | Config for security log destinations. | [waf.securityLogs](#wafsecuritylogs) | No | ### WAF.SecurityLogs |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Turns on the security log. | ``bool`` | No | |``apLogConf`` | The [App Protect WAF log conf](/nic/integrations/app-protect-waf/configuration.md#waf-logs) resource. `apLogConf` references an APLogconf CR by `[/]`. When you start the Ingress Controller with `-plm-storage-url`, PLM must have compiled the referenced APLogConf (`status.bundle.state == ready`). Only works with `apPolicy`. | ``string`` | No | |``apLogBundle`` | The [App Protect WAF log bundle](/nic/integrations/app-protect-waf/configuration.md#waf-bundles) resource. Only works with ``apBundle``. Mutually exclusive with ``apLogBundleSource``. | ``string`` | No | |``apLogBundleSource`` | [Remote source](/nic/integrations/app-protect-waf-v5/bundle-sources.md) for fetching the log profile bundle. Mutually exclusive with ``apLogBundle``. | [waf.apBundleSource](#wafapbundlesource) | No | |``logDest`` | The log destination for the security log. Only accepted variables are ``syslog:server=; localhost; :``, ``stderr``, ````. | ``string`` | No | ### WAF.ApBundleSource The `apBundleSource` object configures how NGINX Ingress Controller fetches a pre-compiled WAF bundle from a remote source. `waf.securityLogs[].apLogBundleSource` uses the same fields. NGINX Ingress Controller supports three source types: - **N1C (NGINX One Console)** -- fetch policies compiled and managed through NGINX One Console. See [policy docs](/nginx-one-console/waf-integration/policy/_index.md). - **NIM (NGINX Instance Manager)** -- fetch policies compiled and managed through NGINX Instance Manager. See [bundle docs](/nim/waf-integration/policies-and-logs/bundles/create-bundle.md). - **HTTPS** -- fetch compiled `.tgz` bundles from any HTTPS server or endpoint. For details and examples, see [Connect F5 WAF for NGINX to bundle sources](/nic/integrations/app-protect-waf-v5/bundle-sources.md). |Field | Description | Type | Required | | ---| ---| ---| --- | |``type`` | Source backend: ``N1C`` (NGINX One Console), ``NIM`` (NGINX Instance Manager), or ``HTTPS``. Defaults to ``HTTPS``. | ``string`` | No | |``url`` | Tenant URL for ``N1C``/``NIM``, or full ``.tgz`` bundle URL for ``HTTPS``. Must use ``https://``. | ``string`` | Yes | |``name`` | Management-plane policy name for ``N1C``/``NIM``. For ``apLogBundleSource``, set this to the log profile name. Ignored for ``HTTPS``. | ``string`` | No | |``namespace`` | Management-plane namespace or tenant. Required for ``N1C``. Not used for ``NIM`` or ``HTTPS``. | ``string`` | No | |``enablePolling`` | Must be explicitly set. When ``true``, NIC re-fetches the bundle at ``pollInterval``. When ``false``, NIC fetches the bundle once at policy creation or update. | ``bool`` | Yes | |``pollInterval`` | How often to re-fetch when ``enablePolling`` is ``true``. Minimum ``1m``, default ``5m``. | ``string`` | No | |``secret`` | Secret in the same namespace as the Policy. For ``N1C``/``NIM``, use ``nginx.com/waf-bundle`` (token or username/password). For ``HTTPS``, use ``kubernetes.io/tls`` for client mTLS (``tls.crt`` and ``tls.key``). | ``string`` | No | |``trustedCertSecret`` | Name of an ``nginx.org/ca`` Secret containing a custom CA certificate (``ca.crt``) for verifying the server TLS certificate. Must be in the same namespace as the Policy. | ``string`` | No | |``insecureSkipVerify`` | Turns off TLS certificate verification. Not recommended for production. | ``bool`` | No | |``verifyChecksum`` | Turns on SHA-256 verification of the downloaded bundle. HTTPS only. | ``bool`` | No | |``timeout`` | Time limit for a single bundle fetch request. Default ``60s``. | ``string`` | No | |``retryAttempts`` | Number of additional fetch attempts after a temporary fetch error (for example, a timeout or HTTP 5xx). Valid range is ``1``-``10``. | ``int`` | No | For example, see the snippets below: for NGINX Instance Manager: ```yaml spec: waf: enable: true apBundleSource: type: NIM url: "https://" name: "" secret: "nim-credentials" enablePolling: true pollInterval: "5m" securityLogs: - enable: true apLogBundleSource: type: NIM url: "https://" name: "" secret: "nim-credentials" enablePolling: true pollInterval: "5m" logDest: "stderr" ``` for NGINX One Console: ```yaml spec: waf: enable: true apBundleSource: type: N1C url: "https://.console.ves.volterra.io" name: "" namespace: "default" secret: "n1c-credentials" enablePolling: true pollInterval: "5m" securityLogs: - enable: true apLogBundleSource: type: N1C url: "https://.console.ves.volterra.io" name: "secops_dashboard" namespace: "default" secret: "n1c-credentials" enablePolling: true pollInterval: "5m" logDest: "stderr" ``` For HTTPS: ```yaml spec: waf: enable: true apBundleSource: url: "https://bundle-server.default.svc.cluster.local/bundles/attack-signatures-blocking.tgz" secret: "bundle-client-tls" trustedCertSecret: "bundle-server-ca" enablePolling: true pollInterval: "5m" securityLogs: - enable: true apLogBundleSource: url: "https://bundle-server.default.svc.cluster.local/bundles/log-default.tgz" secret: "bundle-client-tls" trustedCertSecret: "bundle-server-ca" enablePolling: true pollInterval: "5m" logDest: "stderr" ``` ### WAF merging behavior A VirtualServer or VirtualServerRoute can reference multiple WAF policies, but NGINX Ingress Controller applies only the first one. It ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: waf-policy-one - name: waf-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `waf-policy-one`, and ignores `waf-policy-two`. ## HSTS The HSTS policy sets up [HTTP Strict Transport Security](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security). It instructs browsers to enforce HTTPS connections to the host for a specified duration. For example, the following policy sets an HSTS duration of 30 days and extends the policy to all host subdomains: ```yaml hsts: maxAge: 2592000 includeSubDomains: true ``` When you deploy NGINX Ingress Controller behind a proxy or load balancer that terminates TLS upstream, set `behindProxy` to `true`. In this mode, NGINX uses the `X-Forwarded-Proto` request header to determine whether the connection is HTTPS, instead of checking the `$https` variable directly: ```yaml hsts: maxAge: 2592000 behindProxy: true ``` A VirtualServer that references an HSTS policy must: - Use [TLS termination](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualservertls), or set `behindProxy` to `true` if TLS is terminated upstream. - Reference the policy in the VirtualServer [spec](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserver-specification). You can't reference an HSTS policy in a [route](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroute) or in a VirtualServerRoute [subroute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroutesubroute). If a resource doesn't meet these conditions, NGINX sends status code `500` to clients. **Note:** This feature uses the NGINX `add_header` directive and the [ngx_http_ssl_module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html) `$https` variable. |Field | Description | Type | Required | Default | | ---| ---| ---| --- | --- | |`maxAge` | Sets the duration in seconds that the browser should cache and enforce the HSTS policy. | `int` | Yes | -- | |`includeSubDomains` | Extends the HSTS policy to all subdomains of the host. | `bool` | No | `false` | |`behindProxy` | Sets the HSTS header based on the `X-Forwarded-Proto` request header rather than the `$https` variable. Set this to `true` when you deploy NGINX Ingress Controller behind a proxy or load balancer that terminates TLS upstream. | `bool` | No | `false` | |`preload` | Adds the domain to browsers' [HSTS preload lists](https://hstspreload.org/). Requires `includeSubDomains` to be set to `true` and `maxAge` to be at least 31536000 (one year). | `bool` | No | `false` | **Note:** HSTS instructs browsers to enforce HTTPS for the duration of `maxAge`. Deleting the policy doesn't clear the browser's cached directive, so users may be unable to access the application over HTTP until the cached policy expires. To remove HSTS safely, first set `maxAge` to `0` and apply the updated policy. This instructs browsers to expire the cached directive immediately. Once applied, remove the policy reference from the VirtualServer and delete the policy resource. See the MDN documentation on [HSTS expiration](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Strict-Transport-Security#expiration) for more details. ### HSTS merging behavior A VirtualServer can reference only a single HSTS policy, and NGINX Ingress Controller ignores every subsequent reference. For example, this configuration references two policies: ```yaml policies: - name: hsts-policy-one - name: hsts-policy-two ``` In this example, NGINX Ingress Controller uses the configuration from the first policy reference, `hsts-policy-one`, and ignores `hsts-policy-two`. ## What's next Learn how to [manage Policy resources with kubectl](/nic/configuration/policy-resource/using-policy.md). -------------------------------------------------------------------------------- title: "Reporting resource status" url: https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/reporting-resources-status/index.md -------------------------------------------------------------------------------- This page describes how to view the status of resources managed by F5 NGINX Ingress Controller. ## Ingress resources An Ingress resource status includes the address (an IP address or a DNS name), through which the hosts of that Ingress resource are publicly accessible. You can see the address in the output of the `kubectl get ingress` command, in the ADDRESS column, as shown below: ```shell kubectl get ingresses ``` ```text NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.example.com 12.13.23.123 80, 443 2m ``` NGINX Ingress Controller must be configured to report an Ingress status: 1. Use the command-line flag `-report-ingress-status`. 1. Define a source for an external address. This can be either of: 1. A user defined address, specified in the `external-status-address` ConfigMap key. 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. View the [ConfigMap keys](/nic/configuration/global-configuration/configmap-resource.md) and [Command-line arguments](/nic/configuration/global-configuration/command-line-arguments.md) topics for more information. **Note:** NGINX Ingress Controller does not clear the status of Ingress resources when it is being shut down. ## VirtualServer and VirtualServerRoute resources A VirtualServer or VirtualServerRoute resource includes the status field with information about the state of the resource and the IP address, through which the hosts of that resource are publicly accessible. You can see the status in the output of the `kubectl get virtualservers` or `kubectl get virtualserverroutes` commands as shown below: ```shell kubectl get virtualservers ``` ```text NAME STATE HOST IP PORTS AGE cafe Valid cafe.example.com 12.13.23.123 [80,443] 34s ``` To see an external hostname address associated with a VirtualServer resource, use the `-o wide` option: ```shell kubectl get virtualservers -o wide ``` ```text NAME STATE HOST IP EXTERNALHOSTNAME PORTS AGE cafe Valid cafe.example.com ae430f41a1a0042908655abcdefghijkl-12345678.eu-west-2.elb.amazonaws.com [80,443] 106s ``` **Note:** If there are multiple addresses, only the first one is shown. In order to see additional addresses or extra information about the `Status` of the resource, use the following command: ```shell kubectl describe virtualserver ``` ```text ... Status: External Endpoints: Ip: 12.13.23.123 Ports: [80,443] Message: Configuration for cafe/cafe was added or updated Reason: AddedOrUpdated State: Valid ``` ### Status specification The following fields are reported in both VirtualServer and VirtualServerRoute status: |Field | Description | Type | | ---| ---| --- | |*State* | Current state of the resource. Can be ``Valid``, ``Warning`` an ``Invalid``. For more information, refer to the ``message`` field. | *string* | |*Reason* | The reason of the last update. | *string* | |*Message* | Additional information about the state. | *string* | |*ExternalEndpoints* | A list of external endpoints for which the hosts of the resource are publicly accessible. | *[externalEndpoint](#externalendpoint)* | The *ReferencedBy* field is reported for the VirtualServerRoute status only: |Field | Description | Type | | ---| ---| --- | | *ReferencedBy* | The VirtualServer that references this VirtualServerRoute. Format as ``namespace/name`` | *string* | ### externalEndpoint |Field | Description | Type | | ---| ---| --- | |``IP`` | The external IP address. | ``string`` | |``Hostname`` | The external LoadBalancer Hostname address. | ``string`` | |``Ports`` | A list of external ports. | ``string`` | NGINX Ingress Controller must be configured to report a VirtualServer or VirtualServerRoute status: 1. If you want NGINX Ingress Controller to report the `externalEndpoints`, define a source for an external address (The rest of the fields will be reported without the external address configured). This can be: 1. A user defined address, specified in the `external-status-address` ConfigMap key. 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. View the [ConfigMap keys](/nic/configuration/global-configuration/configmap-resource.md) and [Command-line arguments](/nic/configuration/global-configuration/command-line-arguments.md) topics for more information. **Note:** NGINX Ingress Controller does not clear the status of VirtualServer and VirtualServerRoute resources when it is being shut down. ## Policy resources A Policy resource includes the status field with information about the state of the resource. You can see the status in the output of the `kubectl get policy` command as shown below: ```shell kubectl get policy ``` ```text NAME STATE AGE webapp-policy Valid 30s ``` In order to see additional addresses or extra information about the `Status` of the resource, use the following command: ```shell kubectl describe policy ``` ```text ... Status: Message: Configuration for default/webapp-policy was added or updated Reason: AddedOrUpdated State: Valid ``` ### Status specification The following fields are reported in Policy status: |Field | Description | Type | | ---| ---| --- | |``State`` | Current state of the resource. Can be ``Valid`` or ``Invalid``. For more information, refer to the ``message`` field. | ``string`` | |``Reason`` | The reason of the last update. | ``string`` | |``Message`` | Additional information about the state. | ``string`` | ## TransportServer resources A TransportServer resource includes the status field with information about the state of the resource. You can see the status in the output of the `kubectl get transportserver` command as shown below: ```shell kubectl get transportserver ``` ```text NAME STATE REASON AGE dns-tcp Valid AddedOrUpdated 47m ``` To see additional addresses or extra information about the `Status` of the resource, use the following command: ```shell kubectl describe transportserver ``` ```text Status: Message: Configuration for default/dns-tcp was added or updated Reason: AddedOrUpdated State: Valid ``` ### Status specification The following fields are reported in TransportServer status: |Field | Description | Type | | ---| ---| --- | | *State* | Current state of the resource. Can be ``Valid``, ``Warning`` or ``Invalid``. For more information, refer to the ``message`` field. | *string* | | *Reason* | The reason of the last update. | *string* | | *Message* | Additional information about the state. | *string* | -------------------------------------------------------------------------------- title: "Reporting resource status" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/global-configuration/reporting-resources-status/index.md -------------------------------------------------------------------------------- This page describes how to view the status of resources managed by F5 NGINX Ingress Controller LTS. ## Ingress resources An Ingress resource status includes the address (an IP address or a DNS name), through which the hosts of that Ingress resource are publicly accessible. You can see the address in the output of the `kubectl get ingress` command, in the ADDRESS column, as shown below: ```shell kubectl get ingresses ``` ```text NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.example.com 12.13.23.123 80, 443 2m ``` NGINX Ingress Controller LTS must be configured to report an Ingress status: 1. Use the command-line flag `-report-ingress-status`. 1. Define a source for an external address. This can be either of: 1. A user defined address, specified in the `external-status-address` ConfigMap key. 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. View the [ConfigMap keys](/nic/lts/configuration/global-configuration/configmap-resource.md) and [Command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments.md) topics for more information. **note:** NGINX Ingress Controller LTS does not clear the status of Ingress resources when it is being shut down. ## VirtualServer and VirtualServerRoute resources A VirtualServer or VirtualServerRoute resource includes the status field with information about the state of the resource and the IP address, through which the hosts of that resource are publicly accessible. You can see the status in the output of the `kubectl get virtualservers` or `kubectl get virtualserverroutes` commands as shown below: ```shell kubectl get virtualservers ``` ```text NAME STATE HOST IP PORTS AGE cafe Valid cafe.example.com 12.13.23.123 [80,443] 34s ``` To see an external hostname address associated with a VirtualServer resource, use the `-o wide` option: ```shell kubectl get virtualservers -o wide ``` ```text NAME STATE HOST IP EXTERNALHOSTNAME PORTS AGE cafe Valid cafe.example.com ae430f41a1a0042908655abcdefghijkl-12345678.eu-west-2.elb.amazonaws.com [80,443] 106s ``` **note:** If there are multiple addresses, only the first one is shown. In order to see additional addresses or extra information about the `Status` of the resource, use the following command: ```shell kubectl describe virtualserver ``` ```text ... Status: External Endpoints: Ip: 12.13.23.123 Ports: [80,443] Message: Configuration for cafe/cafe was added or updated Reason: AddedOrUpdated State: Valid ``` ### Status specification The following fields are reported in both VirtualServer and VirtualServerRoute status: |Field | Description | Type | | ---| ---| --- | |*State* | Current state of the resource. Can be ``Valid``, ``Warning`` an ``Invalid``. For more information, refer to the ``message`` field. | *string* | |*Reason* | The reason of the last update. | *string* | |*Message* | Additional information about the state. | *string* | |*ExternalEndpoints* | A list of external endpoints for which the hosts of the resource are publicly accessible. | *[externalEndpoint](#externalendpoint)* | The *ReferencedBy* field is reported for the VirtualServerRoute status only: |Field | Description | Type | | ---| ---| --- | | *ReferencedBy* | The VirtualServer that references this VirtualServerRoute. Format as ``namespace/name`` | *string* | ### externalEndpoint |Field | Description | Type | | ---| ---| --- | |``IP`` | The external IP address. | ``string`` | |``Hostname`` | The external LoadBalancer Hostname address. | ``string`` | |``Ports`` | A list of external ports. | ``string`` | NGINX Ingress Controller LTS must be configured to report a VirtualServer or VirtualServerRoute status: 1. If you want NGINX Ingress Controller LTS to report the `externalEndpoints`, define a source for an external address (The rest of the fields will be reported without the external address configured). This can be: 1. A user defined address, specified in the `external-status-address` ConfigMap key. 1. A Service of the type LoadBalancer configured with an external IP or address and specified by the `-external-service` command-line flag. View the [ConfigMap keys](/nic/lts/configuration/global-configuration/configmap-resource.md) and [Command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments.md) topics for more information. **note:** NGINX Ingress Controller LTS does not clear the status of VirtualServer and VirtualServerRoute resources when it is being shut down. ## Policy resources A Policy resource includes the status field with information about the state of the resource. You can see the status in the output of the `kubectl get policy` command as shown below: ```shell kubectl get policy ``` ```text NAME STATE AGE webapp-policy Valid 30s ``` In order to see additional addresses or extra information about the `Status` of the resource, use the following command: ```shell kubectl describe policy ``` ```text ... Status: Message: Configuration for default/webapp-policy was added or updated Reason: AddedOrUpdated State: Valid ``` ### Status specification The following fields are reported in Policy status: |Field | Description | Type | | ---| ---| --- | |``State`` | Current state of the resource. Can be ``Valid`` or ``Invalid``. For more information, refer to the ``message`` field. | ``string`` | |``Reason`` | The reason of the last update. | ``string`` | |``Message`` | Additional information about the state. | ``string`` | ## TransportServer resources A TransportServer resource includes the status field with information about the state of the resource. You can see the status in the output of the `kubectl get transportserver` command as shown below: ```shell kubectl get transportserver ``` ```text NAME STATE REASON AGE dns-tcp Valid AddedOrUpdated 47m ``` To see additional addresses or extra information about the `Status` of the resource, use the following command: ```shell kubectl describe transportserver ``` ```text Status: Message: Configuration for default/dns-tcp was added or updated Reason: AddedOrUpdated State: Valid ``` ### Status specification The following fields are reported in TransportServer status: |Field | Description | Type | | ---| ---| --- | | *State* | Current state of the resource. Can be ``Valid``, ``Warning`` or ``Invalid``. For more information, refer to the ``message`` field. | *string* | | *Reason* | The reason of the last update. | *string* | | *Message* | Additional information about the state. | *string* | -------------------------------------------------------------------------------- title: "TransportServer resources" url: https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/index.md -------------------------------------------------------------------------------- This document is reference material for the TransportServer resource used by F5 NGINX Ingress Controller. The TransportServer resource allows you to configure TCP, UDP, and TLS Passthrough load balancing. The resource is implemented as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources) for specific use cases. ## Prerequisites - For TCP and UDP, the TransportServer resource must be used in conjunction with the [GlobalConfiguration resource](/nic/configuration/global-configuration/globalconfiguration-resource.md), which must be created separately. - For TLS Passthrough, make sure to enable the [`-enable-tls-passthrough`](/nic/configuration/global-configuration/command-line-arguments#cmdoption-enable-tls-passthrough.md) command-line argument of NGINX Ingress Controller. ## TransportServer Specification The TransportServer resource defines load balancing configuration for TCP, UDP, or TLS Passthrough traffic. Below are a few examples: - TCP load balancing: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: dns-tcp spec: host: dns.example.com listener: name: dns-tcp protocol: TCP tls: secret: cafe-secret upstreams: - name: dns-app service: dns-service port: 5353 action: pass: dns-app ``` - UDP load balancing: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: dns-udp spec: listener: name: dns-udp protocol: UDP upstreams: - name: dns-app service: dns-service port: 5353 upstreamParameters: udpRequests: 1 udpResponses: 1 action: pass: dns-app ``` - TLS passthrough load balancing: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: secure-app spec: listener: name: tls-passthrough protocol: TLS_PASSTHROUGH host: app.example.com upstreams: - name: secure-app service: secure-app port: 8443 action: pass: secure-app ``` | Field | Description | Type | Required | | ---| ---| ---| --- | |``listener`` | The listener on NGINX that will accept incoming connections/datagrams. | [listener](#listener) | Yes | |``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. Wildcard domains like ``*.example.com`` are not allowed. When specified, NGINX will use this host for SNI-based routing. For TLS Passthrough, this field is required. For TCP with TLS termination, specifying the host enables SNI routing and requires specifying a TLS secret.| ``string`` | No | |``tls`` | The TLS termination configuration. Not supported for TLS Passthrough load balancing. | [tls](#tls) | No | |``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | Yes | |``upstreamParameters`` | The upstream parameters. | [upstreamParameters](#upstreamparameters) | No | |``action`` | The action to perform for a client connection/datagram. | [action](#action) | Yes | |``ingressClassName`` | Specifies which Ingress Controller must handle the TransportServer resource. | ``string`` | No | |``streamSnippets`` | Sets a custom snippet in the ``stream`` context. | ``string`` | No | |``serverSnippets`` | Sets a custom snippet in the ``server`` context. | ``string`` | No | \* -- Required for TLS Passthrough load balancing. ### Listener The listener field references a listener that NGINX will use to accept incoming traffic for the TransportServer. For TCP and UDP, the listener must be defined in the [GlobalConfiguration resource](/nic/configuration/global-configuration/globalconfiguration-resource.md). When referencing a listener, both the name and the protocol must match. For TLS Passthrough, use the built-in listener with the name `tls-passthrough` and the protocol `TLS_PASSTHROUGH`. The combination of ``spec.listener.name`` and ``spec.host`` must be unique among all TransportServers. If two TransportServers specify the same combination of ``spec.listener.name`` and ``spec.host``, one of them will be rejected to prevent conflicts. In the case where no host is specified, it is considered an empty string. An example: ```yaml listener: name: dns-udp protocol: UDP ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the listener. | ``string`` | Yes | |``protocol`` | The protocol of the listener. | ``string`` | Yes | ### TLS The tls field defines TLS configuration for a TransportServer. When using TLS termination (not TLS Passthrough), you can specify the host field to enable SNI-based routing, allowing multiple applications to share the same listener port and be distinguished by the TLS SNI hostname. Each application can use its own TLS certificate and key specified via the secret field. ```yaml secret: cafe-secret ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the TransportServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). | ``string`` | No | ### Upstream The upstream defines a destination for the TransportServer. For example: ```yaml name: secure-app service: secure-app port: 8443 maxFails: 3 maxConns: 100 failTimeout: 30s loadBalancingMethod: least_conn ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | |``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. | ``string`` | Yes | |``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. The port must fall into the range ``1..65535``. | ``int`` | Yes | |``maxFails`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_fails) of unsuccessful attempts to communicate with the server that should happen in the duration set by the failTimeout parameter to consider the server unavailable. The default ``1``. | ``int`` | No | |``maxConns`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_conns) of maximum connections to the proxied server. Default value is zero, meaning there is no limit. The default is ``0``. | ``int`` | No | |``failTimeout`` | Sets the [time](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#fail_timeout) during which the specified number of unsuccessful attempts to communicate with the server should happen to consider the server unavailable and the period of time the server will be considered unavailable. The default is ``10s``. | ``string`` | No | |``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [healthcheck](#upstreamhealthcheck) | No | |``loadBalancingMethod`` | The method used to load balance the upstream servers. By default, connections are distributed between the servers using a weighted round-robin balancing method. See the [upstream](http://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) section for available methods and their details. | ``string`` | No | |``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | |``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | ### Upstream.Healthcheck The Healthcheck defines an [active health check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html?#health_check). In the example below we enable a health check for an upstream and configure all the available parameters: ```yaml name: secure-app service: secure-app port: 8443 healthCheck: enable: true interval: 20s timeout: 30s jitter: 3s fails: 5 passes: 5 port: 8080 ``` **Note:** This feature is only supported with NGINX Plus. |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | |``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | |``timeout`` | This overrides the timeout set by [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) which is set in `SessionParameters` for health checks. The default value is ``5s``. | ``string`` | No | |``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | |``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | |``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | |``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | |``match`` | Controls the data to send and the response to expect for the healthcheck. | [match](#upstreamhealthcheckmatch) | No | ### Upstream.Healthcheck.Match The match controls the data to send and the response to expect for the healthcheck: ```yaml match: send: 'GET / HTTP/1.0\r\nHost: localhost\r\n\r\n' expect: "~200 OK" ``` Both `send` and `expect` fields can contain hexadecimal literals with the prefix `\x` followed by two hex digits, for example, `\x80`. See the [match](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) directive for details. |Field | Description | Type | Required | | ---| ---| ---| --- | |``send`` | A string to send to an upstream server. | ``string`` | No | |``expect`` | A literal string or a regular expression that the data obtained from the server should match. The regular expression is specified with the preceding ``~*`` modifier (for case-insensitive matching), or the ``~`` modifier (for case-sensitive matching). NGINX Ingress Controller validates a regular expression using the RE2 syntax. | ``string`` | No | ### UpstreamParameters The upstream parameters define various parameters for the upstreams: ```yaml upstreamParameters: udpRequests: 1 udpResponses: 1 connectTimeout: 60s nextUpstream: true nextUpstreamTimeout: 50s nextUpstreamTries: 1 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``udpRequests`` | The number of datagrams, after receiving which, the next datagram from the same client starts a new session. See the [proxy_requests](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_requests) directive. The default is ``0``. | ``int`` | No | |``udpResponses`` | The number of datagrams expected from the proxied server in response to a client datagram. See the [proxy_responses](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses) directive. By default, the number of datagrams is not limited. | ``int`` | No | |``connectTimeout`` | The timeout for establishing a connection with a proxied server. See the [proxy_connect_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout) directive. The default is ``60s``. | ``string`` | No | |``nextUpstream`` | If a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. See the [proxy_next_upstream](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream) directive. The default is ``true``. | bool | No | |``nextUpstreamTries`` | The number of tries for passing a connection to the next server. See the [proxy_next_upstream_tries](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries) directive. The default is ``0``. | ``int`` | No | |``nextUpstreamTimeout`` | The time allowed to pass a connection to the next server. See the [proxy_next_upstream_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout) directive. The default us ``0``. | ``string`` | No | ### SessionParameters The session parameters define various parameters for TCP connections and UDP sessions. ```yaml sessionParameters: timeout: 50s ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``timeout`` | The timeout between two successive read or write operations on client or proxied server connections. See [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) directive. The default is ``10m``. | ``string`` | No | ### Action The action defines an action to perform for a client connection/datagram. In the example below, client connections/datagrams are passed to an upstream `dns-app`: ```yaml action: pass: dns-app ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``pass`` | Passes connections/datagrams to an upstream. The upstream with that name must be defined in the resource. | ``string`` | Yes | ## Using TransportServer You can use the usual `kubectl` commands to work with TransportServer resources, similar to Ingress resources. For example, the following command creates a TransportServer resource defined in `transport-server-passthrough.yaml` with the name `secure-app`: ```shell kubectl apply -f transport-server-passthrough.yaml ``` ```text transportserver.k8s.nginx.org/secure-app created ``` You can get the resource by running: ```shell kubectl get transportserver secure-app ``` ```text NAME AGE secure-app 46sm ``` In the kubectl get and similar commands, you can also use the short name `ts` instead of `transportserver`. ### Using Snippets Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure [access control](http://nginx.org/en/docs/stream/ngx_stream_access_module.html) in a TransportServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: cafe spec: host: cafe.example.com serverSnippets: | deny 192.168.1.1; allow 192.168.1.0/24; upstreams: - name: tea service: tea-svc port: 80 ``` Snippets can also be specified for a stream. In the example below, we use snippets to [limit the number of connections](https://nginx.org/en/docs/stream/ngx_stream_limit_conn_module.html): ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: cafe spec: host: cafe.example.com streamSnippets: limit_conn_zone $binary_remote_addr zone=addr:10m; serverSnippets: limit_conn addr 1; upstreams: - name: tea service: tea-svc port: 80 ``` **Note:** To configure snippets in the `stream` context, use `stream-snippets` ConfigMap key. For additional information, view the [Advanced configuration with Snippets](/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md) topic. ### Validation Two types of validation are available for the TransportServer resource: - *Structural validation* by the `kubectl` and Kubernetes API server. - *Comprehensive validation* by NGINX Ingress Controller. #### Structural Validation The custom resource definition for the TransportServer includes structural OpenAPI schema which describes the type of every field of the resource. If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: - Example of `kubectl` validation: ```shell kubectl apply -f transport-server-passthrough.yaml ``` ```text error: error validating "transport-server-passthrough.yaml": error validating data: ValidationError(TransportServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.TransportServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f transport-server-passthrough.yaml --validate=false ``` ```text The TransportServer "secure-app" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: spec.upstreams.port in body must be of type integer: "string" ``` If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. #### Comprehensive Validation NGINX Ingress Controller validates the fields of a TransportServer resource. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. You can check if NGINX Ingress Controller successfully applied the configuration for a TransportServer. For our example `secure-app` TransportServer, we can run: ```shell kubectl describe ts secure-app ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/secure-app was added or updated ``` Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a TransportServer `secure-app` with a pass action that references a non-existing upstream, you will get : ```shell kubectl describe ts secure-app ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 2s nginx-ingress-controller TransportServer default/secure-app is invalid and was rejected: spec.action.pass: Not found: "some-app" ``` Note how the events section includes a Warning event with the Rejected reason. **Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. ## Customization via ConfigMap The [ConfigMap](/nic/configuration/global-configuration/configmap-resource.md) keys (except for `stream-snippets`, `stream-log-format`, `resolver-addresses`, `resolver-ipv6`, `resolver-valid` and `resolver-timeout`) do not affect TransportServer resources. -------------------------------------------------------------------------------- title: "TransportServer resources" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/transportserver-resource/index.md -------------------------------------------------------------------------------- This document is reference material for the TransportServer resource used by F5 NGINX Ingress Controller LTS. The TransportServer resource allows you to configure TCP, UDP, and TLS Passthrough load balancing. The resource is implemented as a [Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources) for specific use cases. ## Prerequisites - For TCP and UDP, the TransportServer resource must be used in conjunction with the [GlobalConfiguration resource](/nic/lts/configuration/global-configuration/globalconfiguration-resource.md), which must be created separately. - For TLS Passthrough, make sure to enable the [`-enable-tls-passthrough`](/nic/lts/configuration/global-configuration/command-line-arguments#cmdoption-enable-tls-passthrough.md) command-line argument of NGINX Ingress Controller LTS. ## TransportServer Specification The TransportServer resource defines load balancing configuration for TCP, UDP, or TLS Passthrough traffic. Below are a few examples: - TCP load balancing: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: dns-tcp spec: host: dns.example.com listener: name: dns-tcp protocol: TCP tls: secret: cafe-secret upstreams: - name: dns-app service: dns-service port: 5353 action: pass: dns-app ``` - UDP load balancing: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: dns-udp spec: listener: name: dns-udp protocol: UDP upstreams: - name: dns-app service: dns-service port: 5353 upstreamParameters: udpRequests: 1 udpResponses: 1 action: pass: dns-app ``` - TLS passthrough load balancing: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: secure-app spec: listener: name: tls-passthrough protocol: TLS_PASSTHROUGH host: app.example.com upstreams: - name: secure-app service: secure-app port: 8443 action: pass: secure-app ``` | Field | Description | Type | Required | | ---| ---| ---| --- | |``listener`` | The listener on NGINX that will accept incoming connections/datagrams. | [listener](#listener) | Yes | |``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. Wildcard domains like ``*.example.com`` are not allowed. When specified, NGINX will use this host for SNI-based routing. For TLS Passthrough, this field is required. For TCP with TLS termination, specifying the host enables SNI routing and requires specifying a TLS secret.| ``string`` | No | |``tls`` | The TLS termination configuration. Not supported for TLS Passthrough load balancing. | [tls](#tls) | No | |``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | Yes | |``upstreamParameters`` | The upstream parameters. | [upstreamParameters](#upstreamparameters) | No | |``action`` | The action to perform for a client connection/datagram. | [action](#action) | Yes | |``ingressClassName`` | Specifies which Ingress Controller must handle the TransportServer resource. | ``string`` | No | |``streamSnippets`` | Sets a custom snippet in the ``stream`` context. | ``string`` | No | |``serverSnippets`` | Sets a custom snippet in the ``server`` context. | ``string`` | No | \* -- Required for TLS Passthrough load balancing. ### Listener The listener field references a listener that NGINX will use to accept incoming traffic for the TransportServer. For TCP and UDP, the listener must be defined in the [GlobalConfiguration resource](/nic/lts/configuration/global-configuration/globalconfiguration-resource.md). When referencing a listener, both the name and the protocol must match. For TLS Passthrough, use the built-in listener with the name `tls-passthrough` and the protocol `TLS_PASSTHROUGH`. The combination of ``spec.listener.name`` and ``spec.host`` must be unique among all TransportServers. If two TransportServers specify the same combination of ``spec.listener.name`` and ``spec.host``, one of them will be rejected to prevent conflicts. In the case where no host is specified, it is considered an empty string. An example: ```yaml listener: name: dns-udp protocol: UDP ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the listener. | ``string`` | Yes | |``protocol`` | The protocol of the listener. | ``string`` | Yes | ### TLS The tls field defines TLS configuration for a TransportServer. When using TLS termination (not TLS Passthrough), you can specify the host field to enable SNI-based routing, allowing multiple applications to share the same listener port and be distinguished by the TLS SNI hostname. Each application can use its own TLS certificate and key specified via the secret field. ```yaml secret: cafe-secret ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the TransportServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). | ``string`` | No | ### Upstream The upstream defines a destination for the TransportServer. For example: ```yaml name: secure-app service: secure-app port: 8443 maxFails: 3 maxConns: 100 failTimeout: 30s loadBalancingMethod: least_conn ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | |``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. | ``string`` | Yes | |``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and close client connections/ignore datagrams. The port must fall into the range ``1..65535``. | ``int`` | Yes | |``maxFails`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_fails) of unsuccessful attempts to communicate with the server that should happen in the duration set by the failTimeout parameter to consider the server unavailable. The default ``1``. | ``int`` | No | |``maxConns`` | Sets the [number](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#max_conns) of maximum connections to the proxied server. Default value is zero, meaning there is no limit. The default is ``0``. | ``int`` | No | |``failTimeout`` | Sets the [time](https://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#fail_timeout) during which the specified number of unsuccessful attempts to communicate with the server should happen to consider the server unavailable and the period of time the server will be considered unavailable. The default is ``10s``. | ``string`` | No | |``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check) directive. | [healthcheck](#upstreamhealthcheck) | No | |``loadBalancingMethod`` | The method used to load balance the upstream servers. By default, connections are distributed between the servers using a weighted round-robin balancing method. See the [upstream](http://nginx.org/en/docs/stream/ngx_stream_upstream_module.html#upstream) section for available methods and their details. | ``string`` | No | |``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | |``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | ### Upstream.Healthcheck The Healthcheck defines an [active health check](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html?#health_check). In the example below we enable a health check for an upstream and configure all the available parameters: ```yaml name: secure-app service: secure-app port: 8443 healthCheck: enable: true interval: 20s timeout: 30s jitter: 3s fails: 5 passes: 5 port: 8080 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | |``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | |``timeout`` | This overrides the timeout set by [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) which is set in `SessionParameters` for health checks. The default value is ``5s``. | ``string`` | No | |``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | |``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | |``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | |``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | |``match`` | Controls the data to send and the response to expect for the healthcheck. | [match](#upstreamhealthcheckmatch) | No | ### Upstream.Healthcheck.Match The match controls the data to send and the response to expect for the healthcheck: ```yaml match: send: 'GET / HTTP/1.0\r\nHost: localhost\r\n\r\n' expect: "~200 OK" ``` Both `send` and `expect` fields can contain hexadecimal literals with the prefix `\x` followed by two hex digits, for example, `\x80`. See the [match](https://nginx.org/en/docs/stream/ngx_stream_upstream_hc_module.html#match) directive for details. |Field | Description | Type | Required | | ---| ---| ---| --- | |``send`` | A string to send to an upstream server. | ``string`` | No | |``expect`` | A literal string or a regular expression that the data obtained from the server should match. The regular expression is specified with the preceding ``~*`` modifier (for case-insensitive matching), or the ``~`` modifier (for case-sensitive matching). NGINX Ingress Controller LTS validates a regular expression using the RE2 syntax. | ``string`` | No | ### UpstreamParameters The upstream parameters define various parameters for the upstreams: ```yaml upstreamParameters: udpRequests: 1 udpResponses: 1 connectTimeout: 60s nextUpstream: true nextUpstreamTimeout: 50s nextUpstreamTries: 1 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``udpRequests`` | The number of datagrams, after receiving which, the next datagram from the same client starts a new session. See the [proxy_requests](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_requests) directive. The default is ``0``. | ``int`` | No | |``udpResponses`` | The number of datagrams expected from the proxied server in response to a client datagram. See the [proxy_responses](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses) directive. By default, the number of datagrams is not limited. | ``int`` | No | |``connectTimeout`` | The timeout for establishing a connection with a proxied server. See the [proxy_connect_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout) directive. The default is ``60s``. | ``string`` | No | |``nextUpstream`` | If a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. See the [proxy_next_upstream](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream) directive. The default is ``true``. | bool | No | |``nextUpstreamTries`` | The number of tries for passing a connection to the next server. See the [proxy_next_upstream_tries](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries) directive. The default is ``0``. | ``int`` | No | |``nextUpstreamTimeout`` | The time allowed to pass a connection to the next server. See the [proxy_next_upstream_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout) directive. The default us ``0``. | ``string`` | No | ### SessionParameters The session parameters define various parameters for TCP connections and UDP sessions. ```yaml sessionParameters: timeout: 50s ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``timeout`` | The timeout between two successive read or write operations on client or proxied server connections. See [proxy_timeout](http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout) directive. The default is ``10m``. | ``string`` | No | ### Action The action defines an action to perform for a client connection/datagram. In the example below, client connections/datagrams are passed to an upstream `dns-app`: ```yaml action: pass: dns-app ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``pass`` | Passes connections/datagrams to an upstream. The upstream with that name must be defined in the resource. | ``string`` | Yes | ## Using TransportServer You can use the usual `kubectl` commands to work with TransportServer resources, similar to Ingress resources. For example, the following command creates a TransportServer resource defined in `transport-server-passthrough.yaml` with the name `secure-app`: ```shell kubectl apply -f transport-server-passthrough.yaml ``` ```text transportserver.k8s.nginx.org/secure-app created ``` You can get the resource by running: ```shell kubectl get transportserver secure-app ``` ```text NAME AGE secure-app 46sm ``` In the kubectl get and similar commands, you can also use the short name `ts` instead of `transportserver`. ### Using Snippets Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure [access control](http://nginx.org/en/docs/stream/ngx_stream_access_module.html) in a TransportServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: cafe spec: host: cafe.example.com serverSnippets: | deny 192.168.1.1; allow 192.168.1.0/24; upstreams: - name: tea service: tea-svc port: 80 ``` Snippets can also be specified for a stream. In the example below, we use snippets to [limit the number of connections](https://nginx.org/en/docs/stream/ngx_stream_limit_conn_module.html): ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: cafe spec: host: cafe.example.com streamSnippets: limit_conn_zone $binary_remote_addr zone=addr:10m; serverSnippets: limit_conn addr 1; upstreams: - name: tea service: tea-svc port: 80 ``` **note:** To configure snippets in the `stream` context, use `stream-snippets` ConfigMap key. For additional information, view the [Advanced configuration with Snippets](/nic/lts/configuration/ingress-resources/advanced-configuration-with-snippets.md) topic. ### Validation Two types of validation are available for the TransportServer resource: - *Structural validation* by the `kubectl` and Kubernetes API server. - *Comprehensive validation* by NGINX Ingress Controller LTS. #### Structural Validation The custom resource definition for the TransportServer includes structural OpenAPI schema which describes the type of every field of the resource. If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: - Example of `kubectl` validation: ```shell kubectl apply -f transport-server-passthrough.yaml ``` ```text error: error validating "transport-server-passthrough.yaml": error validating data: ValidationError(TransportServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.TransportServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f transport-server-passthrough.yaml --validate=false ``` ```text The TransportServer "secure-app" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: spec.upstreams.port in body must be of type integer: "string" ``` If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller LTS will validate it further. #### Comprehensive Validation NGINX Ingress Controller LTS validates the fields of a TransportServer resource. If a resource is invalid, NGINX Ingress Controller LTS will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller LTS will ignore it. You can check if NGINX Ingress Controller LTS successfully applied the configuration for a TransportServer. For our example `secure-app` TransportServer, we can run: ```shell kubectl describe ts secure-app ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 3s nginx-ingress-controller Configuration for default/secure-app was added or updated ``` Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid resource, NGINX Ingress Controller LTS will reject it and emit a Rejected event. For example, if you create a TransportServer `secure-app` with a pass action that references a non-existing upstream, you will get : ```shell kubectl describe ts secure-app ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 2s nginx-ingress-controller TransportServer default/secure-app is invalid and was rejected: spec.action.pass: Not found: "some-app" ``` Note how the events section includes a Warning event with the Rejected reason. **Note**: If you make an existing resource invalid, NGINX Ingress Controller LTS will reject it and remove the corresponding configuration from NGINX. ## Customization via ConfigMap The [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource.md) keys (except for `stream-snippets`, `stream-log-format`, `resolver-addresses`, `resolver-ipv6`, `resolver-valid` and `resolver-timeout`) do not affect TransportServer resources. -------------------------------------------------------------------------------- title: "Troubleshooting VirtualServer resources" url: https://docs.nginx.com/nginx-ingress-controller/lts/troubleshooting/troubleshoot-virtualserver/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot VirtualServer and VirtualServer resource events. ## Inspecting VirtualServer and VirtualServerRoute resource events After creating or updating a VirtualServer resource, you can immediately check if the NGINX configuration for that resource was successfully by using `kubectl describe vs `: ```shell kubectl describe vs cafe ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated ``` In the above example, we have a `Normal` event with the `AddedOrUpdate` reason, which informs us that the configuration was successfully applied. Checking the events of a VirtualServerRoute is similar: ```shell kubectl describe vsr coffee ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 1m nginx-ingress-controller Configuration for default/coffee was added or updated ``` -------------------------------------------------------------------------------- title: "Troubleshooting VirtualServer resources" url: https://docs.nginx.com/nginx-ingress-controller/troubleshooting/troubleshoot-virtualserver/index.md -------------------------------------------------------------------------------- This page describes how to troubleshoot VirtualServer and VirtualServer resource events. ## Inspecting VirtualServer and VirtualServerRoute resource events After creating or updating a VirtualServer resource, you can immediately check if the NGINX configuration for that resource was successfully by using `kubectl describe vs `: ```shell kubectl describe vs cafe ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated ``` In the above example, we have a `Normal` event with the `AddedOrUpdate` reason, which informs us that the configuration was successfully applied. Checking the events of a VirtualServerRoute is similar: ```shell kubectl describe vsr coffee ``` ```shell Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 1m nginx-ingress-controller Configuration for default/coffee was added or updated ``` -------------------------------------------------------------------------------- title: "Build NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/install/build/index.md -------------------------------------------------------------------------------- This document describes how to build an F5 NGINX Ingress Controller image from source code and upload it to a private Docker registry. It also includes information on the Makefile targets and variables. **Note:** If you do not need to build a custom image, see the [pre-built image options](#pre-built-images) at the end of this guide. ## Before you begin To get started, you need the following software installed on your machine: - [Docker v19.03 or higher](https://docs.docker.com/engine/release-notes/19.03/) - [GNU Make](https://www.gnu.org/software/make/) - [git](https://git-scm.com/) - [OpenSSL](https://www.openssl.org/), optionally, if you would like to generate a self-signed certificate and a key for the default server. - For NGINX Plus users, download the certificate (_nginx-repo.crt_) and key (_nginx-repo.key_) from [MyF5](https://my.f5.com). Although NGINX Ingress Controller is written in Golang, you don't need to have Golang installed. You can download the precompiled binary file or build NGINX Ingress Controller in a Docker container. ## Prepare the environment {#prepare-environment} Get your system ready for building and pushing the NGINX Ingress Controller image. 1. Sign in to your private registry. Replace `` with the path to your own private registry. ```shell docker login ``` 2. Clone the NGINX Ingress Controller GitHub repository. Replace `` with the version of NGINX Ingress Controller you want. ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch cd kubernetes-ingress ``` For instance if you want to clone version v, the commands to run would be: ```shell git clone https://github.com/nginx/kubernetes-ingress.git --branch v cd kubernetes-ingress ``` ## Build the NGINX Ingress Controller image {#build-image} After setting up your environment, follow these steps to build the NGINX Ingress Controller image. **Note:** If you have a local Golang environment and want to build the binary yourself, remove `TARGET=download` from the make commands. If you don't have Golang but still want to build the binary, use `TARGET=container`. ### For NGINX Build the image. Replace `` with your private registry's path. - For a Debian-based image: ```shell make debian-image PREFIX=/nginx-ingress TARGET=download ``` - For an Alpine-based image: ```shell make alpine-image PREFIX=/nginx-ingress TARGET=download ``` **What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](#makefile-details). This version number is used for tracking and deployment purposes. ### For NGINX Plus Place your NGINX Plus license files (_nginx-repo.crt_ and _nginx-repo.key_) in the project's root folder. To verify they're in place, run: ```shell ls nginx-repo.* ``` You should see: ```text nginx-repo.crt nginx-repo.key ``` Build the image. Replace `` with your private registry's path. ```shell make debian-image-plus PREFIX=/nginx-plus-ingress TARGET=download ``` **What to expect**: The image is built and tagged with a version number, which is derived from the `VERSION` variable in the [_Makefile_](#makefile-details). This version number is used for tracking and deployment purposes. **Note:** If a patch for NGINX Plus is released, make sure to rebuild your image to get the latest version. If your system is caching the Docker layers and not updating the packages, add `DOCKER_BUILD_OPTIONS="--pull --no-cache"` to the make command. ## Push the image to your private registry {#push-image} Once you've successfully built the NGINX or NGINX Plus Ingress Controller image, the next step is to upload it to your private Docker registry. This makes the image available for deployment to your Kubernetes cluster. ### For NGINX Upload the NGINX image. If you're using a custom tag, append `TAG=your-tag` to the command. Replace `` with your private registry's path. ```shell make push PREFIX=/nginx-ingress ``` ### For NGINX Plus Upload the NGINX Plus image. Like with the NGINX image, if you're using a custom tag, add `TAG=your-tag` to the end of the command. Replace `` with your private registry's path. ```shell make push PREFIX=/nginx-plus-ingress ``` ## Makefile details {#makefile-details} This section provides comprehensive information on the targets and variables available in the _Makefile_. These targets and variables allow you to customize how you build, tag, and push your NGINX or NGINX Plus images. ### Key Makefile targets {#key-makefile-targets} **Note:** To view available _Makefile_ targets, run `make` with no target or type `make help`. Key targets include: | Target | Description | |---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | _build_ | Creates the NGINX Ingress Controller binary with your local Go environment. | | _alpine-image_ | Builds an Alpine-based image with NGINX. | | _alpine-image-plus_ | Builds an Alpine-based image with NGINX Plus. | | _alpine-image-plus-fips_ | Builds an Alpine-based image with NGINX Plus and FIPS. | | _alpine-image-nap-v5-plus-fips_ | Builds an Alpine-based image with NGINX Plus, the [F5 WAF for NGINX v5](/nginx-app-protect/) module and FIPS. | | _debian-image_ | Builds a Debian-based image with NGINX. | | _debian-image-plus_ | Builds a Debian-based image with NGINX Plus. | | _debian-image-nap-plus_ | Builds a Debian-based image with NGINX Plus and the [F5 WAF for NGINX](/nginx-app-protect/) module. | | _debian-image-nap-v5-plus_ | Builds a Debian-based image with NGINX Plus and the [F5 WAF for NGINX v5](/nginx-app-protect/) module. | | _debian-image-dos-plus_ | Builds a Debian-based image with NGINX Plus and the [F5 DoS for NGINX](/nginx-app-protect-dos/) module. | | _debian-image-nap-dos-plus_ | Builds a Debian-based image with NGINX Plus, [F5 WAF for NGINX](/nginx-app-protect/) and [F5 DoS for NGINX](/nginx-app-protect-dos/) modules. | | _ubi-image_ | Builds a UBI-based image with NGINX for [OpenShift](https://www.openshift.com/) clusters. | | _ubi-image-plus_ | Builds a UBI-based image with NGINX Plus for [OpenShift](https://www.openshift.com/) clusters. | | _ubi-image-nap-plus_ | Builds a UBI-based image with NGINX Plus and the [F5 WAF for NGINX](/nginx-app-protect/) module for [OpenShift](https://www.openshift.com/) clusters. | | _ubi-image-nap-v5-plus_ | Builds a UBI-based image with NGINX Plus and the [F5 WAF for NGINX v5](/nginx-app-protect/) module for [OpenShift](https://www.openshift.com/) clusters. | | _ubi-image-dos-plus_ | Builds a UBI-based image with NGINX Plus and the [F5 DoS for NGINX](/nginx-app-protect-dos/) module for [OpenShift](https://www.openshift.com/) clusters. | | _ubi-image-nap-dos-plus_ |

Builds a UBI-based image with NGINX Plus, [F5 WAF for NGINX](/nginx-app-protect/) and the [F5 DoS for NGINX](/nginx-app-protect-dos/) module for [OpenShift](https://www.openshift.com/) clusters.| **Note:** For RHEL, save your organization and activation keys in a file named _rhel_license_ at the project root. Ensure they are on separate lines, such as: - RHEL_ORGANIZATION=1111111 - RHEL_ACTIVATION_KEY=your-key ### Additional useful targets {#other-makefile-targets} A few other useful targets: | Target | Description | |---------------------------------------|---------------| | _push_ | Pushes the built image to the Docker registry. Configures with `PREFIX` and `TAG`. | | _all_ | Runs `test`, `lint`, `verify-codegen`, `update-crds`, and `debian-image`. Stops and reports an error if any of these targets fail. | | _test_ | Runs unit tests. | ### Makefile variables you can customize {#makefile-variables} The _Makefile_ includes several key variables. You have the option to either modify these variables directly in the _Makefile_ or override them when you run the `make` command. | Variable | Description | |-----------------------------------------|---------------| | _ARCH_ | Defines the architecture for the image and binary. The default is `amd64`, but you can also use `arm64`. | | _PREFIX_ | Gives the image its name. The default is `nginx/nginx-ingress`. | | _TAG_ | Adds a tag to the image. This is often the version of NGINX Ingress Controller. | | _DOCKER\_BUILD\_OPTIONS_ | Allows for additional [options](https://docs.docker.com/engine/reference/commandline/build/#options) during the `docker build` process, like `--pull`. | | _TARGET_ |

Determines the build environment. NGINX Ingress Controller compiles locally in a Golang environment by default. Ensure the NGINX Ingress Controller repo resides in your `$GOPATH` if you select this option.

Alternatively, you can set `TARGET=container` to build using a Docker [Golang](https://hub.docker.com/_/golang/) container. To skip compiling the binary if you're on a specific tag or the latest `main` branch commit, set `TARGET=download`.

| ## Alternatives to building your own image {#pre-built-images} If you prefer not to build your own NGINX Ingress Controller image, you can use pre-built images. Here are your options: **NGINX Ingress Controller**: Download the image `nginx/nginx-ingress` from [DockerHub](https://hub.docker.com/r/nginx/nginx-ingress) or [GitHub](https://github.com/nginx/kubernetes-ingress/pkgs/container/kubernetes-ingress). **NGINX Plus Ingress Controller**: You have two options for this: - Download the image using your NGINX Ingress Controller subscription certificate and key. View the [Download NGINX Ingress Controller from the F5 Registry](/nic/install/images/registry-download.md) topic. - Use your NGINX Ingress Controller subscription JWT token to get the image. View the [Add an NGINX Ingress Controller image to your cluster](/nic/install/images/add-image-to-cluster.md) topic. -------------------------------------------------------------------------------- title: "Enable OpenTracing (Removed in v5.0.0)" url: https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/opentracing/index.md -------------------------------------------------------------------------------- This topic describes how to OpenTracing with F5 NGINX Ingress Controller. NGINX Ingress Controller supports [OpenTracing](https://opentracing.io/) with the third-party module [opentracing-contrib/nginx-opentracing](https://github.com/opentracing-contrib/nginx-opentracing). **Note:** OpenTracing support has been removed from v5.0.0 of NGINX Ingress Controller. From v5.1.0 onwards, you should follow the guidance in [Configure OpenTelemetry](/nic/logging-and-monitoring/opentelemetry.md). ## Prerequisites 1. Use a NGINX Ingress Controller image that contains OpenTracing. - You can find the images that include OpenTracing listed [in the technical specs doc](/nic/technical-specifications.md#supported-docker-images). - Alternatively, you follow [Build NGINX Ingress Controller](/nic/install/build.md) using `debian-image` (or `alpine-image`) for NGINX or `debian-image-plus` (or `alpine-image-plus`) for NGINX Plus. - [Jaeger](https://github.com/jaegertracing/jaeger-client-cpp), [Zipkin](https://github.com/rnburn/zipkin-cpp-opentracing) and [Datadog](https://github.com/DataDog/dd-opentracing-cpp/) tracers are installed by default. 1. Enable snippets annotations by setting the [`enable-snippets`](/nic/configuration/global-configuration/command-line-arguments#cmdoption-enable-snippets) command-line argument to true. 1. Load the OpenTracing module. You need to load the module with the configuration for the chosen tracer using the following ConfigMap keys: - `opentracing-tracer`: sets the path to the vendor tracer binary plugin. This is the path you used in the COPY line of step *ii* above. - `opentracing-tracer-config`: sets the tracer configuration in JSON format. The following example shows how to use these two keys to load the module with Jaeger tracer: ```yaml opentracing-tracer: "/usr/local/lib/libjaegertracing_plugin.so" opentracing-tracer-config: | { "service_name": "nginx-ingress", "propagation_format": "w3c", "sampler": { "type": "const", "param": 1 }, "reporter": { "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831" } } ``` ## Enable OpenTracing globally To enable OpenTracing globally (for all Ingress, VirtualServer and VirtualServerRoute resources), set the `opentracing` ConfigMap key to `True`: ```yaml opentracing: True ``` ## Enable or disable OpenTracing per Ingress resource You can use annotations to enable or disable OpenTracing for a specific Ingress resource. As mentioned in the prerequisites section, both `opentracing-tracer` and `opentracing-tracer-config` must be configured. Consider the following two cases: ### OpenTracing is globally disabled 1. To enable OpenTracing for a specific Ingress resource, use the server snippet annotation: ```yaml nginx.org/server-snippets: | opentracing on; ``` 1. To enable OpenTracing for specific paths: - You need to use [Mergeable Ingress resources](/nic/configuration/ingress-resources/cross-namespace-configuration) - You need to use the location snippets annotation to enable OpenTracing for the paths of a specific Minion Ingress resource: ```yaml nginx.org/location-snippets: | opentracing on; ``` ### OpenTracing is globally enabled 1. To disable OpenTracing for a specific Ingress resource, use the server snippet annotation: ```yaml nginx.org/server-snippets: | opentracing off; ``` 1. To disable OpenTracing for specific paths: - You need to use [Mergeable Ingress resources](/nic/configuration/ingress-resources/cross-namespace-configuration) - You need to use the location snippets annotation to disable OpenTracing for the paths of a specific Minion Ingress resource: ```yaml nginx.org/location-snippets: | opentracing off; ``` ## Customize OpenTracing You can customize OpenTracing through the supported [OpenTracing module directives](https://github.com/opentracing-contrib/nginx-opentracing/blob/master/doc/Reference.md). Use the location-snippets ConfigMap keys or annotations to insert those directives into the generated NGINX configuration. For example, to propagate the active span context for upstream requests, you need to set the `opentracing_propagate_context` directive, which you can add to an Ingress resource using the location snippets annotation: ```yaml nginx.org/location-snippets: | opentracing_propagate_context; ``` **Note:** The `opentracing_propagate_context` and `opentracing_grpc_propagate_context` directives can be used in `http`, `server` or `location` contexts according to the [module documentation](https://github.com/opentracing-contrib/nginx-opentracing/blob/master/doc/Reference.md#opentracing_propagate_context). However, because of the way the module works and how NGINX Ingress Controller generates the NGINX configuration, it is only possible to use the directive in the `location` context. -------------------------------------------------------------------------------- title: "Manage Policy resources with kubectl" url: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/using-policy/index.md -------------------------------------------------------------------------------- Use standard `kubectl` commands to work with Policy resources, just as you would with built-in Kubernetes resources. For example, the following command creates a Policy resource defined in `access-control-policy-allow.yaml` with the name `webapp-policy`: ```shell kubectl apply -f access-control-policy-allow.yaml policy.k8s.nginx.org/webapp-policy configured ``` Get the resource by running: ```shell kubectl get policy webapp-policy NAME AGE webapp-policy 27m ``` For `kubectl get` and similar commands, you can also use the short name `pol` instead of `policy`. ## Attach policies to a resource You can apply policies to VirtualServer, VirtualServerRoute, and Ingress resources. For example: - VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe spec: host: cafe.example.com tls: secret: cafe-secret policies: # spec policies - name: policy1 upstreams: - name: coffee service: coffee-svc port: 80 routes: - path: /tea policies: # route policies - name: policy2 namespace: cafe route: tea/tea - path: /coffee policies: # route policies - name: policy3 namespace: cafe action: pass: coffee ``` For VirtualServer, you can apply a policy: * to all routes (spec policies) * to a specific route (route policies) Route policies of the same type override spec policies. In the example above, if `policy-1` and `policy-3` are both `accessControl` policies, NGINX applies `policy-3` to requests for `cafe.example.com/coffee`. NGINX enforces this override: the spec policies apply in the `server` context of the configuration, and the route policies apply in the `location` context. As a result, the route policies of the same type take precedence. - VirtualServerRoute, referenced by the VirtualServer above: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: tea namespace: tea spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 subroutes: # subroute policies - path: /tea policies: - name: policy4 namespace: tea action: pass: tea ``` For VirtualServerRoute, you can apply a policy to a subroute (subroute policies). Subroute policies of the same type override spec policies. In the example above, if `policy-1` (in the VirtualServer) and `policy-4` are both `accessControl` policies, NGINX applies `policy-4` to requests for `cafe.example.com/tea`. As with the VirtualServer, NGINX enforces this override. Subroute policies always override route policies, regardless of type. For example, NGINX Ingress Controller ignores `policy-2` from the VirtualServer route for the `/tea` subroute, because the subroute has its own policies, `policy4` in this case. If the subroute had no policies, NGINX Ingress Controller would apply `policy-2` instead. NGINX Ingress Controller enforces this override: the `location` context for the subroute has either route policies or subroute policies, but never both. - Ingress: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: nginx.org/policies: "webapp-policy" spec: ingressClassName: nginx tls: - hosts: - cafe.example.com secretName: tls-secret rules: - host: cafe.example.com http: paths: - path: /tea pathType: Prefix backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: Prefix backend: service: name: coffee-svc port: number: 80 ``` For Ingress, you can apply policies: * to a single Ingress * to a master Ingress, where minion Ingresses inherit the policies * to minion Ingresses, where minion policies override master policies ## Invalid policies NGINX treats a policy as invalid if any of the following conditions is true: - The policy doesn't pass [comprehensive validation](#comprehensive-validation). - The policy isn't present in the cluster. - The policy doesn't meet its type-specific requirements. For example, an `ingressMTLS` policy requires TLS termination turned on in the VirtualServer. For an invalid policy, NGINX returns the 500 status code for client requests, following these rules: - If a policy is referenced in a VirtualServer `route` or a VirtualServerRoute `subroute`, NGINX returns the 500 status code for requests to the URIs of that route or subroute. - If a policy is referenced in the VirtualServer `spec`, NGINX returns the 500 status code for requests to all URIs of that VirtualServer. If a policy is invalid, the VirtualServer or VirtualServerRoute gets the [status](/nic/configuration/global-configuration/reporting-resources-status.md#virtualserver-and-virtualserverroute-resources) state `Warning`, with a message that explains why the policy is invalid. ## Validation Two types of validation are available for the Policy resource: - *Structural validation*, done by `kubectl` and the Kubernetes API server. - *Comprehensive validation*, done by NGINX Ingress Controller. ### Structural validation The custom resource definition for the Policy includes a structural OpenAPI schema, which describes the type of every field of the resource. If you try to create or update a resource that violates the structural schema, for example, if the resource uses a string value instead of an array of strings in the `allow` field, `kubectl` and the Kubernetes API server reject the resource. - Example of `kubectl` validation: ```shell kubectl apply -f access-control-policy-allow.yaml error: error validating "access-control-policy-allow.yaml": error validating data: ValidationError(Policy.spec.accessControl.allow): invalid type for org.nginx.k8s.v1.Policy.spec.accessControl.allow: got "string", expected "array"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f access-control-policy-allow.yaml --validate=false The Policy "webapp-policy" is invalid: spec.accessControl.allow: Invalid value: "string": spec.accessControl.allow in body must be of type array: "string" ``` If a resource passes structural validation, NGINX Ingress Controller's comprehensive validation runs next. ### Comprehensive validation NGINX Ingress Controller validates the fields of a Policy resource. If a resource is invalid, NGINX Ingress Controller rejects it. The resource continues to exist in the cluster, but NGINX Ingress Controller ignores it. Use `kubectl` to check whether NGINX Ingress Controller successfully applied a Policy configuration. For the example `webapp-policy` Policy, run: ```shell kubectl describe pol webapp-policy . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 11s nginx-ingress-controller Policy default/webapp-policy was added or updated ``` The events section includes a Normal event with the AddedOrUpdated reason, which tells you the configuration applied successfully. If you create an invalid resource, NGINX Ingress Controller rejects it and emits a Rejected event. For example, if you create a Policy `webapp-policy` with an invalid IP `10.0.0.` in the `allow` field, you get: ```shell kubectl describe policy webapp-policy . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 7s nginx-ingress-controller Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP ``` The events section includes a Warning event with the Rejected reason. This information is also available in the `status` field of the Policy resource. Note the Status section of the Policy: ```shell kubectl describe pol webapp-policy . . . Status: Message: Policy default/webapp-policy is invalid and was rejected: spec.accessControl.allow[0]: Invalid value: "10.0.0.": must be a CIDR or IP Reason: Rejected State: Invalid ``` **Note:** If you make an existing resource invalid, NGINX Ingress Controller rejects it. -------------------------------------------------------------------------------- title: "VirtualServer and VirtualServerRoute resources" url: https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/index.md -------------------------------------------------------------------------------- This document is reference material for the VirtualServer and VirtualServerRoute resources used by F5 NGINX Ingress Controller. VirtualServer and VirtualServerRoute resources are load balancing configurations recommended as an alternative to the Ingress resource. They enable use cases not supported with the Ingress resource, such as traffic splitting and advanced content-based routing. The resources are implemented as [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources) for specific use cases. --- ## VirtualServer specification The VirtualServer resource defines load balancing configuration for a domain name, such as `example.com`. The example below shows a VirtualServer with both traditional static routes and a dynamic route using `routeSelector`. The route with path `/dynamic` will automatically include any VirtualServerRoute resources that have the label `app: cafe`: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe spec: host: cafe.example.com listener: http: http-8083 https: https-8443 tls: secret: cafe-secret gunzip: on upstreams: - name: tea service: tea-svc port: 80 - name: coffee service: coffee-svc port: 80 routes: - path: /tea action: pass: tea - path: /coffee action: pass: coffee - path: ~ ^/decaf/.*\\.jpg$ action: pass: coffee - path: =/green/tea action: pass: tea - path: /dynamic routeSelector: matchLabels: app: cafe ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. The ``host`` value needs to be unique among all Ingress and VirtualServer resources. See also [Handling Host and Listener Collisions](/nic/configuration/host-and-listener-collisions.md). | ``string`` | Yes | |``listener`` | Sets a custom HTTP and/or HTTPS listener. Valid fields are `listener.http` and `listener.https`. Each field must reference the name of a valid listener defined in a GlobalConfiguration resource | [listener](#virtualserverlistener) | No | |``tls`` | The TLS termination configuration. | [tls](#virtualservertls) | No | |``gunzip`` | Enables or disables [decompression](/nginx/admin-guide/web-server/compression.md) of gzipped responses for clients. Allowed values “on”/“off”, “true”/“false” or “yes”/“no”. If the ``gunzip`` value is not set, it defaults to ``off``. | ``boolean`` | No | |``externalDNS`` | The externalDNS configuration for a VirtualServer. | [externalDNS](#virtualserverexternaldns) | No | |``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer. | ``string`` | No | |``policies`` | A list of policies. | [[]policy](#virtualserverpolicy) | No | |``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | |``routes`` | A list of routes. | [[]route](#virtualserverroute) | No | |``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServer resource. | ``string`` | No | |``internalRoute`` | Specifies if the VirtualServer resource is an internal route or not. | ``boolean`` | No | |``http-snippets`` | Sets a custom snippet in the http context. | ``string`` | No | |``server-snippets`` | Sets a custom snippet in server context. Overrides the ``server-snippets`` ConfigMap key. | ``string`` | No | |``add-header-inherit`` | Controls header inheritance behavior at the server level. Allowed values are ``on``, ``off``, and ``merge``. | ``string`` | No | ### VirtualServer.TLS The tls field defines TLS configuration for a VirtualServer. For example: ```yaml secret: cafe-secret redirect: enable: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the VirtualServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the host of the VirtualServer. If the secret is not specified but [wildcard TLS secret](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-wildcard-tls-secret) is configured, NGINX will use the wildcard secret for TLS termination. | ``string`` | No | |``redirect`` | The redirect configuration of the TLS for a VirtualServer. | [tls.redirect](#virtualservertlsredirect) | No | ### VirtualServer.TLS.Redirect | |``cert-manager`` | The cert-manager configuration of the TLS for a VirtualServer. | [tls.cert-manager](#virtualservertlscertmanager) | No | ### VirtualServer.TLS.CertManager | ### VirtualServer.TLS.Redirect The redirect field configures a TLS redirect for a VirtualServer: ```yaml enable: true code: 301 basedOn: scheme ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables a TLS redirect for a VirtualServer. The default is ``False``. | ``boolean`` | No | |``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | |``basedOn`` | The attribute of a request that NGINX will evaluate to send a redirect. The allowed values are ``scheme`` (the scheme of the request) or ``x-forwarded-proto`` (the ``X-Forwarded-Proto`` header of the request). The default is ``scheme``. | ``string`` | No | ### VirtualServer.Policy | ### VirtualServer.TLS.CertManager The cert-manager field configures x509 automated Certificate management for VirtualServer resources using cert-manager (cert-manager.io). Please see the [cert-manager configuration documentation](https://cert-manager.io/docs/configuration/) for more information on deploying and configuring Issuers. Example: ```yaml cert-manager: cluster-issuer: "my-issuer-name" ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``issuer`` | the name of an Issuer. An Issuer is a cert-manager resource which describes the certificate authority capable of signing certificates. The Issuer must be in the same namespace as the VirtualServer resource. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | |``cluster-issuer`` | the name of a ClusterIssuer. A ClusterIssuer is a cert-manager resource which describes the certificate authority capable of signing certificates. It does not matter which namespace your VirtualServer resides, as ClusterIssuers are non-namespaced resources. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | |``issuer-kind`` | The kind of the external issuer resource, for example AWSPCAIssuer. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | |``issuer-group`` | The API group of the external issuer controller, for example awspca.cert-manager.io. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | |``common-name`` | This field allows you to configure spec.commonName for the Certificate to be generated. This configuration adds a CN to the x509 certificate. | ``string`` | No | |``duration`` | This field allows you to configure spec.duration field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | |``renew-before`` | this annotation allows you to configure spec.renewBefore field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | |``usages`` | This field allows you to configure spec.usages field for the Certificate to be generated. Pass a string with comma-separated values i.e. ``key agreement,digital signature, server auth``. An exhaustive list of supported key usages can be found in the [the cert-manager api documentation](https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.KeyUsage). | ``string`` | No | |``issue-temp-cert`` | When ``true``, ask cert-manager for a [temporary self-signed certificate](https://cert-manager.io/docs/usage/certificate/#temporary-certificates-while-issuing) pending the issuance of the Certificate. This allows HTTPS-only servers to use ACME HTTP01 challenges when the TLS secret does not exist yet. | ``boolean`` | No | ### VirtualServer.Listener The listener field defines a custom HTTP and/or HTTPS listener. The respective listeners used must reference the name of a listener defined using a [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) resource. For example: ```yaml http: http-8083 https: https-8443 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``http`` | The name of am HTTP listener defined in a [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) resource. | ``string`` | No | |``https`` | The name of an HTTPS listener defined in a [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource.md) resource. | ``string`` | No | ### VirtualServer.ExternalDNS The externalDNS field configures controlling DNS records dynamically for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Please see the [ExternalDNS configuration documentation](https://kubernetes-sigs.github.io/external-dns/) for more information on deploying and configuring ExternalDNS and Providers. Example: ```yaml enable: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables ExternalDNS integration for a VirtualServer resource. The default is ``false``. | ``string`` | No | |``labels`` | Configure labels to be applied to the Endpoint resources that will be consumed by ExternalDNS. | ``map[string]string`` | No | |``providerSpecific`` | Configure provider specific properties which holds the name and value of a configuration which is specific to individual DNS providers. | [[]ProviderSpecific](#virtualserverexternaldnsproviderspecific) | No | |``recordTTL`` | TTL for the DNS record. This defaults to 0 if not defined. See [the ExternalDNS TTL documentation for provider-specific defaults](https://kubernetes-sigs.github.io/external-dns/v0.14.2/ttl/#providers) | ``int64`` | No | |``recordType`` | The record Type that should be created, e.g. "A", "AAAA", "CNAME". This is automatically computed based on the external endpoints if not defined. | ``string`` | No | ### VirtualServer.ExternalDNS.ProviderSpecific The providerSpecific field of the externalDNS block allows the specification of provider specific properties which is a list of key value pairs of configurations which are specific to individual DNS providers. Example: ```yaml - name: my-name value: my-value - name: my-name2 value: my-value2 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the key value pair. | ``string`` | Yes | |``value`` | The value of the key value pair. | ``string`` | Yes | ### VirtualServer.Policy The policy field references a [Policy resource](/nic/configuration/policy-resource/before-you-begin.md) by its name and optional namespace. For example: ```yaml name: access-control ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of a policy. If the policy doesn't exist or invalid, NGINX will respond with an error response with the `500` status code. | ``string`` | Yes | |``namespace`` | The namespace of a policy. If not specified, the namespace of the VirtualServer resource is used. | ``string`` | No | ### VirtualServer.Route The route defines rules for matching client requests to actions like passing a request to an upstream. For example: ```yaml path: /tea action: pass: tea ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``path`` | The path of the route. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), a longest prefix match ( ``^~/images/`` ) that prevents regex evaluation, an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix (must start with ``/`` ), a longest prefix match (must start with ``^~`` ) or an exact match (must start with ``=`` ), the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all routes of the VirtualServer. See [Path matching](/nic/configuration/path-matching.md) for details on matching types and priority. Check the [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive for more information. | ``string`` | Yes | |``policies`` | A list of policies. The policies override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying policies](/nic/configuration/policy-resource/applying-policies.md) for more details. | [[]policy](#virtualserverpolicy) | No | |``action`` | The default action to perform for a request. | [action](#action) | No | |``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer route. | ``string`` | No | |``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | |``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | |``route`` | The name of a VirtualServerRoute resource that defines this route. If the VirtualServerRoute belongs to a different namespace than the VirtualServer, you need to include the namespace. For example, ``tea-namespace/tea``. | ``string`` | No | |``routeSelector`` | The RouteSelector allows selecting VirtualServerRoute resources using label selectors instead of specifying them by name and namespace. | [routeSelector](#virtualserverrouterouteselector) | No | |``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | |``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` ConfigMap key. | ``string`` | No | |``add-header-inherit`` | Controls header inheritance behavior at the location level. Allowed values are ``on``, ``off``, and ``merge``. | ``string`` | No | \* -- a route must include exactly one of the following: `action`, `splits`, `route` or `routeSelector` ### VirtualServer.Route.RouteSelector The RouteSelector field allows you to dynamically select VirtualServerRoute resources using label selectors instead of specifying them by name and namespace. This provides a more flexible and scalable way to associate VirtualServerRoutes with a VirtualServer route. See the [VirtualServerRoute specification](#virtualserverroute-specification) section below for examples of how to use RouteSelector with VirtualServerRoute resources. |Field | Description | Type | Required | | ---| ---| ---| --- | |``matchLabels`` | A map of key-value pairs. Each key-value pair in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. | ``map[string]string`` | Yes | |``matchExpressions`` | A list of label selector requirements. The requirements are ANDed. For more information on label selector requirements, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). | N/A | No | ## VirtualServerRoute specification The VirtualServerRoute resource defines a route for a VirtualServer. It can consist of one or multiple subroutes. The VirtualServerRoute is an alternative to [Mergeable Ingress types](/nic/configuration/ingress-resources/cross-namespace-configuration.md). VirtualServer routes can reference VirtualServerRoute resources in two ways: by name using the `route` field, or dynamically using the `routeSelector` field with label selectors. The `routeSelector` approach allows you to add new VirtualServerRoute resources without modifying the VirtualServer configuration. #### Standard In this example, the VirtualServer `cafe` from the namespace `cafe-ns` defines a route that references a specific VirtualServerRoute by name. VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe-ns spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 routes: - path: /tea action: pass: tea - path: /coffee route: coffee-ns/coffee ``` VirtualServerRoute: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: coffee namespace: coffee-ns spec: host: cafe.example.com upstreams: - name: latte service: latte-svc port: 80 - name: espresso service: espresso-svc port: 80 subroutes: - path: /coffee/latte action: pass: latte - path: /coffee/espresso action: pass: espresso ``` #### RouteSelector In this example, the VirtualServer `cafe` from the namespace `cafe-ns` uses `routeSelector` to dynamically select any VirtualServerRoute with the label `app: coffee`. VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe-ns spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 routes: - path: /tea action: pass: tea - path: /decaf routeSelector: matchLabels: app: coffee ``` VirtualServerRoute: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: decaf namespace: coffee-ns labels: app: coffee spec: host: cafe.example.com upstreams: - name: latte service: latte-svc port: 80 - name: espresso service: espresso-svc port: 80 subroutes: - path: /decaf/latte action: pass: latte - path: /decaf/espresso action: pass: espresso ``` Note that each subroute must have a `path` that starts with the same prefix (here `/coffee`), which is defined in the route of the VirtualServer. Additionally, the `host` in the VirtualServerRoute must be the same as the `host` of the VirtualServer. |Field | Description | Type | Required | | ---| ---| ---| --- | |``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. Must be the same as the ``host`` of the VirtualServer that references this resource. | ``string`` | Yes | |``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | |``subroutes`` | A list of subroutes. | [[]subroute](#virtualserverroutesubroute) | No | |``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServerRoute resource. Must be the same as the ``ingressClassName`` of the VirtualServer that references this resource. | ``string``_ | No | ### VirtualServerRoute.Subroute The subroute defines rules for matching client requests to actions like passing a request to an upstream. For example: ```yaml path: /coffee action: pass: coffee ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``path`` | The path of the subroute. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), a longest prefix match ( ``^~/images/`` ) that prevents regex evaluation, an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix, the path must start with the same path as the path of the route of the VirtualServer that references this resource. In the case of a longest prefix match, the path must start with the same path as the path of the route of the VirtualServer that references this resource, including the ``^~`` modifier. In the case of an exact or regex match, the path must be the same as the path of the route of the VirtualServer that references this resource. In the case of regex matches, multiple VirtualServer regex routes (``~`` or ``~*``) can reference the same VirtualServerRoute; the subroutes must form an exact set match with those VirtualServer paths. See [Multiple regex routes in a VirtualServerRoute](#multiple-regex-routes-in-a-virtualserverroute) for details. A matching path of the route of the VirtualServer but in different type is not accepted, e.g. a regex path (`~/match`) cannot be used with a prefix path in VirtualServer (`/match`). In the case of a prefix, longest prefix match or an exact match, the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. Whitespace between the regex modifier and the path is stripped before comparison, so ``~/api`` and ``~ /api`` are treated as the same path. The path must be unique among the paths of all subroutes of the VirtualServerRoute after normalization. See [Path matching](/nic/configuration/path-matching.md) for details on matching types, priority, and subroute constraints. | ``string`` | Yes | |``policies`` | A list of policies. The policies override *all* policies defined in the route of the VirtualServer that references this resource. The policies also override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying policies](/nic/configuration/policy-resource/applying-policies.md) for more details. | [[]policy](#virtualserverpolicy) | No | |``action`` | The default action to perform for a request. | [action](#action) | No | |``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServerRoute subroute. | ``string`` | No | |``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | |``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | |``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | |``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` of the VirtualServer (if set) or the ``location-snippets`` ConfigMap key. | ``string`` | No | |``add-header-inherit`` | Controls header inheritance behavior at the location level. Allowed values are ``on``, ``off``, and ``merge``. | ``string`` | No | \* -- a subroute must include exactly one of the following: `action` or `splits`. ## Common VirtualServer and VirtualServerRoute specifications ### Upstream The upstream defines a destination for the routing configuration. For example: ```yaml name: tea service: tea-svc subselector: version: canary port: 80 lb-method: round_robin fail-timeout: 10s max-fails: 1 max-conns: 32 keepalive: 32 connect-timeout: 30s read-timeout: 30s send-timeout: 30s next-upstream: "error timeout non_idempotent" next-upstream-timeout: 5s next-upstream-tries: 10 client-max-body-size: 2m tls: enable: true ``` **Note**: The WebSocket protocol is supported without any additional configuration. |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | |``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). If the Service belongs to a different namespace than the VirtualServer or VirtualServerRoute, you need to include the namespace. For example, ``tea-namespace/tea``. If the service doesn't exist, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. For NGINX Plus only, services of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) are also supported in the same namespace (check the [prerequisites](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services#prerequisites) ). | ``string`` | Yes | |``subselector`` | Selects the pods within the service using label keys and values. By default, all pods of the service are selected. Note: the specified labels are expected to be present in the pods when they are created. If the pod labels are updated, NGINX Ingress Controller will not see that change until the number of the pods is changed. | ``map[string]string`` | No | |``use-cluster-ip`` | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | ``boolean`` | No | |``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. The port must fall into the range ``1..65535``. | ``uint16`` | Yes | |``lb-method`` | The load [balancing method](/nginx/admin-guide/load-balancer/http-load-balancer.md#choosing-a-load-balancing-method). To use the round-robin method, specify ``round_robin``. The default is specified in the ``lb-method`` ConfigMap key. | ``string`` | No | |``fail-timeout`` | The time during which the specified number of unsuccessful attempts to communicate with an upstream server should happen to consider the server unavailable. See the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the server directive. The default is set in the ``fail-timeout`` ConfigMap key. | ``string`` | No | |``max-fails`` | The number of unsuccessful attempts to communicate with an upstream server that should happen in the duration set by the ``fail-timeout`` to consider the server unavailable. See the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the server directive. The default is set in the ``max-fails`` ConfigMap key. | ``int`` | No | |``max-conns`` | The maximum number of simultaneous active connections to an upstream server. See the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the server directive. By default there is no limit. Note: if keepalive connections are enabled, the total number of active and idle keepalive connections to an upstream server may exceed the ``max_conns`` value. | ``int`` | No | |``keepalive`` | Configures the cache for connections to upstream servers. The value ``0`` disables the cache. See the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. The default is set in the ``keepalive`` ConfigMap key. | ``int`` | No | |``connect-timeout`` | The timeout for establishing a connection with an upstream server. See the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) directive. The default is specified in the ``proxy-connect-timeout`` ConfigMap key. | ``string`` | No | |``read-timeout`` | The timeout for reading a response from an upstream server. See the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) directive. The default is specified in the ``proxy-read-timeout`` ConfigMap key. | ``string`` | No | |``send-timeout`` | The timeout for transmitting a request to an upstream server. See the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) directive. The default is specified in the ``proxy-send-timeout`` ConfigMap key. | ``string`` | No | |``next-upstream`` | Specifies in which cases a request should be passed to the next upstream server. See the [proxy_next_upstream](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) directive. The default is ``error timeout``. | ``string`` | No | |``next-upstream-timeout`` | The time during which a request can be passed to the next upstream server. See the [proxy_next_upstream_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. The ``0`` value turns off the time limit. The default is ``0``. | ``string`` | No | |``next-upstream-tries`` | The number of possible tries for passing a request to the next upstream server. See the [proxy_next_upstream_tries](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. The ``0`` value turns off this limit. The default is ``0``. | ``int`` | No | |``client-max-body-size`` | Sets the maximum allowed size of the client request body. See the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. The default is set in the ``client-max-body-size`` ConfigMap key. | ``string`` | No | |``client-body-buffer-size`` | Sets the value of the [client_body_buffer_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) directive. The default is set in the ``client-body-buffer-size`` ConfigMap key. | ``string`` | No | |``tls`` | The TLS configuration for the Upstream. | [tls](#upstreamtls) | No | |``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [healthcheck](#upstreamhealthcheck) | No | |``slow-start`` | The slow start allows an upstream server to gradually recover its weight from 0 to its nominal value after it has been recovered or became available or when the server becomes available after a period of time it was considered unavailable. By default, the slow start is disabled. See the [slow_start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start) parameter of the server directive. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods and will be ignored. | ``string`` | No | |``queue`` | Configures a queue for an upstream. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request. By default, no queue is configured. Note: this feature is supported only in NGINX Plus. | [queue](#upstreamqueue) | No | |``buffering`` | Enables buffering of responses from the upstream server. See the [proxy_buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) directive. The default is set in the ``proxy-buffering`` ConfigMap key. | ``boolean`` | No | |``buffers`` | Configures the buffers used for reading a response from the upstream server for a single connection. | [buffers](#upstreambuffers) | No | |``buffer-size`` | Sets the size of the buffer used for reading the first part of a response received from the upstream server. See the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) directive. The default is set in the ``proxy-buffer-size`` ConfigMap key. | ``string`` | No | |``busy-buffers-size`` | Sets the size of the buffer used for reading a response from the upstream server when the response is larger than the ``buffer-size``. See the [proxy_busy_buffers_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) directive. The default is set in the ``proxy-busy-buffers-size`` ConfigMap key. | ``string`` | No | |``ntlm`` | Allows proxying requests with NTLM Authentication. See the [ntlm](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ntlm) directive. In order for NTLM authentication to work, it is necessary to enable keepalive connections to upstream servers using the ``keepalive`` field. Note: this feature is supported only in NGINX Plus.| ``boolean`` | No | |``type`` |The type of the upstream. Supported values are ``http`` and ``grpc``. The default is ``http``. For gRPC, it is necessary to enable HTTP/2 in the [ConfigMap](/nic/configuration/global-configuration/configmap-resource.md#listeners) and configure TLS termination in the VirtualServer. | ``string`` | No | |``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | |``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | ### Upstream.Buffers The buffers field configures the buffers used for reading a response from the upstream server for a single connection: ```yaml number: 4 size: 8K ``` See the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive for additional information. |Field | Description | Type | Required | | ---| ---| ---| --- | |``number`` | Configures the number of buffers. The default is set in the ``proxy-buffers`` ConfigMap key. | ``int`` | Yes | |``size`` | Configures the size of a buffer. The default is set in the ``proxy-buffers`` ConfigMap key. | ``string`` | Yes | ### Upstream.TLS |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables HTTPS for requests to upstream servers. The default is ``False`` , meaning that HTTP will be used. Note: by default, NGINX will not verify the upstream server certificate. To enable the verification, configure an [EgressMTLS Policy](/nic/configuration/policy-resource/policy-reference.md#egressmtls). | ``boolean`` | No | ### Upstream.Queue The queue field configures a queue. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request: ```yaml size: 10 timeout: 60s ``` See [`queue`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue) directive for additional information. Note: This feature is supported only in NGINX Plus. |Field | Description | Type | Required | | ---| ---| ---| --- | |``size`` | The size of the queue. | ``int`` | Yes | |``timeout`` | The timeout of the queue. A request cannot be queued for a period longer than the timeout. The default is ``60s``. | ``string`` | No | ### Upstream.Healthcheck The Healthcheck defines an [active health check](/nginx/admin-guide/load-balancer.md#http-health-check). In the example below we enable a health check for an upstream and configure all the available parameters, including the `slow-start` parameter combined with [`mandatory` and `persistent`](/nginx/admin-guide/load-balancer/http-health-check.md#mandatory-health-checks): ```yaml name: tea service: tea-svc port: 80 slow-start: 30s healthCheck: enable: true path: /healthz interval: 20s jitter: 3s fails: 5 passes: 5 port: 8080 tls: enable: true connect-timeout: 10s read-timeout: 10s send-timeout: 10s headers: - name: Host value: my.service statusMatch: "! 500" mandatory: true persistent: true keepalive-time: 60s ``` **Note:** This feature is supported only in NGINX Plus. |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | |``path`` | The path used for health check requests. The default is ``/``. This not configurable for gRPC type upstreams. | ``string`` | No | |``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | |``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | |``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | |``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | |``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | |``tls`` | The TLS configuration used for health check requests. By default, the ``tls`` field of the upstream is used. | [upstream.tls](#upstreamtls) | No | |``connect-timeout`` | The timeout for establishing a connection with an upstream server. By default, the ``connect-timeout`` of the upstream is used. | ``string`` | No | |``read-timeout`` | The timeout for reading a response from an upstream server. By default, the ``read-timeout`` of the upstream is used. | ``string`` | No | |``send-timeout`` | The timeout for transmitting a request to an upstream server. By default, the ``send-timeout`` of the upstream is used. | ``string`` | No | |``headers`` | The request headers used for health check requests. NGINX Plus always sets the ``Host`` , ``User-Agent`` and ``Connection`` headers for health check requests. | [[]header](#header) | No | |``statusMatch`` | The expected response status codes of a health check. By default, the response should have status code 2xx or 3xx. Examples: ``"200"`` , ``"! 500"`` , ``"301-303 307"``. See the documentation of the [match](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html?#match) directive. This not supported for gRPC type upstreams. | ``string`` | No | |``grpcStatus`` | The expected [gRPC status code](https://github.com/grpc/grpc/blob/master/doc/statuscodes.md#status-codes-and-their-use-in-grpc) of the upstream server response to the [Check method](https://github.com/grpc/grpc/blob/master/doc/health-checking.md). Configure this field only if your gRPC services do not implement the gRPC health checking protocol. For example, configure ``12`` if the upstream server responds with `12 (UNIMPLEMENTED)` status code. Only valid on gRPC type upstreams. | ``int`` | No | |``grpcService`` | The gRPC service to be monitored on the upstream server. Only valid on gRPC type upstreams. | ``string`` | No | |``mandatory`` | Require every newly added server to pass all configured health checks before NGINX Plus sends traffic to it. If this is not specified, or is set to false, the server will be initially considered healthy. When combined with [slow-start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start), it gives a new server more time to connect to databases and “warm up” before being asked to handle their full share of traffic. | ``bool`` | No | |``persistent`` | Set the initial “up” state for a server after reload if the server was considered healthy before reload. Enabling persistent requires that the mandatory parameter is also set to `true`. | ``bool`` | No | |``keepalive-time`` | Enables [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) connections for health checks and specifies the time during which requests can be processed through one keepalive connection. The default is ``60s``. | ``string`` | No | ### Upstream.SessionCookie The SessionCookie field configures session persistence which allows requests from the same client to be passed to the same upstream server. The information about the designated upstream server is passed in a session cookie generated by NGINX. In the example below, we configure session persistence with a session cookie for an upstream and configure all the available parameters: ```yaml name: tea service: tea-svc port: 80 sessionCookie: enable: true name: srv_id path: / expires: 1h domain: .example.com httpOnly: false secure: true samesite: strict ``` See the [`sticky`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html?#sticky) directive for additional information. The session cookie corresponds to the `sticky cookie` method. **Note:** Session persistence via sticky cookie is available in both NGINX (since version 1.29.6) and NGINX Plus. |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables session persistence with a session cookie for an upstream server. The default is ``false``. | ``boolean`` | No | |``name`` | The name of the cookie. | ``string`` | Yes | |``path`` | The path for which the cookie is set. | ``string`` | No | |``expires`` | The time for which a browser should keep the cookie. Can be set to the special value ``max`` , which will cause the cookie to expire on ``31 Dec 2037 23:55:55 GMT``. | ``string`` | No | |``domain`` | The domain for which the cookie is set. | ``string`` | No | |``httpOnly`` | Adds the ``HttpOnly`` attribute to the cookie. | ``boolean`` | No | |``secure`` | Adds the ``Secure`` attribute to the cookie. | ``boolean`` | No | |``samesite`` | Adds the ``SameSite`` attribute to the cookie. The allowed values are: ``strict``, ``lax``, ``none`` | ``string`` | No | ### Action The action defines an action to perform for a request. In the example below, client requests are passed to an upstream `coffee`: ```yaml path: /coffee action: pass: coffee ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``pass`` | Passes requests to an upstream. The upstream with that name must be defined in the resource. | ``string`` | No | |``redirect`` | Redirects requests to a provided URL. | [action.redirect](#actionredirect) | No | |``return`` | Returns a preconfigured response. | [action.return](#actionreturn) | No | |``proxy`` | Passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). | [action.proxy](#actionproxy) | No | \* -- an action must include exactly one of the following: `pass`, `redirect`, `return` or `proxy`. ### Action.Redirect The redirect action defines a redirect to return for a request. In the example below, client requests are passed to a url `http://www.nginx.com`: ```yaml redirect: url: http://www.nginx.com code: 301 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` , ``$http_x_forwarded_proto`` , ``$request_uri`` , ``$host``. Variables must be enclosed in curly braces. For example: ``${host}${request_uri}``. | ``string`` | Yes | |``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | ### Action.Return The return action defines a preconfigured response for a request. In the example below, NGINX will respond with the preconfigured response for every request: ```yaml return: code: 200 type: text/plain body: "Hello World\n" headers: - name: x-coffee value: espresso ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``code`` | The status code of the response. The allowed values are: ``2XX``, ``4XX`` or ``5XX``. The default is ``200``. | ``int`` | No | |``type`` | The MIME type of the response. The default is ``text/plain``. | ``string`` | No | |``body`` | The body of the response. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``Request is ${request_uri}\n``. | ``string`` | Yes | |``headers`` | The custom headers of the response. | [[]Action.Return.Header](#actionreturnheader) | No | \* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing` and `$connections_waiting`. ### Action.Return.Header The header defines an HTTP Header for a canned response in an actionReturn: ```yaml name: x-coffee value: espresso ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. | ``string`` | Yes | ### Action.Proxy The proxy action passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). In the example below, the request URI is rewritten to `/`, and the request and the response headers are modified: ```yaml proxy: upstream: coffee requestHeaders: pass: true set: - name: My-Header value: Value - name: Client-Cert value: ${ssl_client_escaped_cert} responseHeaders: add: - name: My-Header value: Value - name: IC-Nginx-Version value: ${nginx_version} always: true hide: - x-internal-version ignore: - Expires - Set-Cookie pass: - Server rewritePath: / ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``upstream`` | The name of the upstream which the requests will be proxied to. The upstream with that name must be defined in the resource. | ``string`` | Yes | |``requestHeaders`` | The request headers modifications. | [action.Proxy.RequestHeaders](#actionproxyrequestheaders) | No | |``responseHeaders`` | The response headers modifications. | [action.Proxy.ResponseHeaders](#actionproxyresponseheaders) | No | |``rewritePath`` | The rewritten URI. If the route path is a regular expression -- starts with `~` -- the `rewritePath` can include capture groups with ``$1-9``. For example `$1` for the first group, and so on. For more information, check the [rewrite](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/rewrites) example. | ``string`` | No | ### Action.Proxy.RequestHeaders The RequestHeaders field modifies the headers of the request to the proxied upstream server. |Field | Description | Type | Required | | ---| ---| ---| --- | |``pass`` | Passes the original request headers to the proxied upstream server. See the [proxy_pass_request_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers) directive for more information. Default is true. | ``bool`` | No | |``set`` | Allows redefining or appending fields to present request headers passed to the proxied upstream servers. See the [proxy_set_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive for more information. | [[]header](#actionproxyrequestheaderssetheader) | No | ### Action.Proxy.RequestHeaders.Set.Header The header defines an HTTP Header: ```yaml name: My-Header value: My-Value ``` It is possible to override the default value of the `Host` header, which NGINX Ingress Controller sets to [`$host`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_host): ```yaml name: Host value: example.com ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | \* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` (NGINX Plus only) and `$jwt_header_` (NGINX Plus only). ### Action.Proxy.ResponseHeaders The ResponseHeaders field modifies the headers of the response to the client. |Field | Description | Type | Required | | ---| ---| ---| --- | |``hide`` | The headers that will not be passed* in the response to the client from a proxied upstream server. See the [proxy_hide_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directive for more information. | ``[]string`` | No | |``pass`` | Allows passing the hidden header fields* to the client from a proxied upstream server. See the [proxy_pass_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directive for more information. | ``[]string`` | No | |``ignore`` | Disables processing of certain headers** to the client from a proxied upstream server. See the [proxy_ignore_headers](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers) directive for more information. | ``[]string`` | No | |``add`` | Adds headers to the response to the client. | [[]addHeader](#addheader) | No | \* -- Default hidden headers are: `Date`, `Server`, `X-Pad` and `X-Accel-...`. \** -- The following fields can be ignored: `X-Accel-Redirect`, `X-Accel-Expires`, `X-Accel-Limit-Rate`, `X-Accel-Buffering`, `X-Accel-Charset`, `Expires`, `Cache-Control`, `Set-Cookie` and `Vary`. ### AddHeader The addHeader defines an HTTP Header with an optional `always` field: ```yaml name: My-Header value: My-Value always: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | |``always`` | If set to true, add the header regardless of the response status code**. Default is false. See the [add_header](http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header) directive for more information. | ``bool`` | No | \* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` (NGINX Plus only) and `$jwt_header_` (NGINX Plus only). **Note:** If `always` is false, the response header is added only if the response status code is any of `200`, `201`, `204`, `206`, `301`, `302`, `303`, `304`, `307` or `308`. ### Split The split defines a weight for an action as part of the splits configuration. In the example below NGINX passes 80% of requests to the upstream `coffee-v1` and the remaining 20% to `coffee-v2`: ```yaml splits: - weight: 80 action: pass: coffee-v1 - weight: 20 action: pass: coffee-v2 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``weight`` | The weight of an action. Must fall into the range ``0..100``. The sum of the weights of all splits must be equal to ``100``. | ``int`` | Yes | |``action`` | The action to perform for a request. | [action](#action) | Yes | ### Match The match defines a match between conditions and an action or splits. In the example below, NGINX routes requests with the path `/coffee` to different upstreams based on the value of the cookie `user`: - `user=john` -> `coffee-future` - `user=bob` -> `coffee-deprecated` - If the cookie is not set or not equal to either `john` or `bob`, NGINX routes to `coffee-stable` ```yaml path: /coffee matches: - conditions: - cookie: user value: john action: pass: coffee-future - conditions: - cookie: user value: bob action: pass: coffee-deprecated action: pass: coffee-stable ``` In the next example, NGINX routes requests based on the value of the built-in [`$request_method` variable](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_method), which represents the HTTP method of a request: - all POST requests -> `coffee-post` - all non-POST requests -> `coffee` ```yaml path: /coffee matches: - conditions: - variable: $request_method value: POST action: pass: coffee-post action: pass: coffee ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``conditions`` | A list of conditions. Must include at least 1 condition. | [[]condition](#condition) | Yes | |``action`` | The action to perform for a request. | [action](#action) | No | |``splits`` | The splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | **Note:** A match must include exactly one of the following: `action` or `splits`. ### Condition The condition defines a condition in a match. |Field | Description | Type | Required | | ---| ---| ---| --- | |``header`` | The name of a header. Must consist of alphanumeric characters or ``-``. | ``string`` | No | |``cookie`` | The name of a cookie. Must consist of alphanumeric characters or ``_``. | ``string`` | No | |``argument`` | The name of an argument. Must consist of alphanumeric characters or ``_``. | ``string`` | No | |``variable`` | The name of an NGINX variable. Must start with ``$``. See the list of the supported variables below the table. | ``string`` | No | |``value`` | The value to match the condition against. How to define a value is shown below the table. | ``string`` | Yes | **Note:** a condition must include exactly one of the following: `header`, `cookie`, `argument` or `variable`. Supported NGINX variables: `$args`, `$http2`, `$https`, `$remote_addr`, `$remote_port`, `$query_string`, `$request`, `$request_body`, `$request_uri`, `$request_method`, `$scheme`. Find the documentation for each variable [here](https://nginx.org/en/docs/varindex.html). The value supports two kinds of matching: - *Case-insensitive string comparison*. For example: - `john` -- case-insensitive matching that succeeds for strings, such as `john`, `John`, `JOHN`. - `!john` -- negation of the case-insensitive matching for john that succeeds for strings, such as `bob`, `anything`, `''` (empty string). - *Matching with a regular expression*. Note that NGINX supports regular expressions compatible with those used by the Perl programming language (PCRE). For example: - `~^yes` -- a case-sensitive regular expression that matches any string that starts with `yes`. For example: `yes`, `yes123`. - `!~^yes` -- negation of the previous regular expression that succeeds for strings like `YES`, `Yes123`, `noyes`. (The negation mechanism is not part of the PCRE syntax). - `~*no$` -- a case-insensitive regular expression that matches any string that ends with `no`. For example: `no`, `123no`, `123NO`. **Note:** A value must not include any unescaped double quotes (`"`) and must not end with an unescaped backslash (`\`). For example, the following are invalid values: `some"value`, `somevalue\`. ### ErrorPage The errorPage defines a custom response for a route for the case when either an upstream server responds with (or NGINX generates) an error status code. The custom response can be a redirect or a canned response. See the [error_page](https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page) directive for more information. ```yaml path: /coffee errorPages: - codes: [502, 503] redirect: code: 301 url: https://nginx.org - codes: [404] return: code: 200 body: "Original resource not found, but success!" ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``codes`` | A list of error status codes. | ``[]int`` | Yes | |``redirect`` | The redirect action for the given status codes. | [errorPage.Redirect](#errorpageredirect) | No | |``return`` | The canned response action for the given status codes. | [errorPage.Return](#errorpagereturn) | No | **Note:** An errorPage must include exactly one of the following: `return` or `redirect`. ### ErrorPage.Redirect The redirect defines a redirect for an errorPage. In the example below, NGINX responds with a redirect when a response from an upstream server has a 404 status code. ```yaml codes: [404] redirect: code: 301 url: ${scheme}://cafe.example.com/error.html ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | |``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` and ``$http_x_forwarded_proto``. Variables must be enclosed in curly braces. For example: ``${scheme}``. | ``string`` | Yes | ### ErrorPage.Return The return defines a canned response for an errorPage. In the example below, NGINX responds with a canned response when a response from an upstream server has either 401 or 403 status code. ```yaml codes: [401, 403] return: code: 200 type: application/json body: | {\"msg\": \"You don't have permission to do this\"} headers: - name: x-debug-original-statuses value: ${upstream_status} ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``code`` | The status code of the response. The default is the status code of the original response. | ``int`` | No | |``type`` | The MIME type of the response. The default is ``text/html``. | ``string`` | No | |``body`` | The body of the response. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | Yes | |``headers`` | The custom headers of the response. | [[]errorPage.Return.Header](#errorpagereturnheader) | No | ### ErrorPage.Return.Header The header defines an HTTP Header for a canned response in an errorPage: ```yaml name: x-debug-original-statuses value: ${upstream_status} ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | No | ## Using VirtualServer and VirtualServerRoute You can use the usual `kubectl` commands to work with VirtualServer and VirtualServerRoute resources, similar to Ingress resources. For example, the following command creates a VirtualServer resource defined in `cafe-virtual-server.yaml` with the name `cafe`: ```shell kubectl apply -f cafe-virtual-server.yaml ``` ```text virtualserver.k8s.nginx.org "cafe" created ``` You can get the resource by running: ```shell kubectl get virtualserver cafe ``` ```text NAME STATE HOST IP PORTS AGE cafe Valid cafe.example.com 12.13.23.123 [80,443] 3m ``` In `kubectl get` and similar commands, you can use the short name `vs` instead of `virtualserver`. Similarly, for VirtualServerRoute you can use `virtualserverroute` or the short name `vsr`. ### Using Snippets Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure several NGINX features in a VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe spec: http-snippets: | limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s; proxy_cache_path /tmp keys_zone=one:10m; host: cafe.example.com tls: secret: cafe-secret server-snippets: | limit_req zone=mylimit burst=20; upstreams: - name: tea service: tea-svc port: 80 - name: coffee service: coffee-svc port: 80 routes: - path: /tea location-snippets: | proxy_cache one; proxy_cache_valid 200 10m; action: pass: tea - path: /coffee action: pass: coffee ``` For additional information, view the [Advanced configuration with Snippets](/nic/configuration/ingress-resources/advanced-configuration-with-snippets.md) topic. ### Validation Two types of validation are available for VirtualServer and VirtualServerRoute resources: - *Structural validation* by the `kubectl` and Kubernetes API server. - *Comprehensive validation* by NGINX Ingress Controller. #### Structural Validation The custom resource definitions for VirtualServer and VirtualServerRoute include structural OpenAPI schema which describes the type of every field of those resources. If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: - Example of `kubectl` validation: ```shell kubectl apply -f cafe-virtual-server.yaml ``` ```text error: error validating "cafe-virtual-server.yaml": error validating data: ValidationError(VirtualServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.VirtualServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f cafe-virtual-server.yaml --validate=false ``` ```text The VirtualServer "cafe" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: spec.upstreams.port in body must be of type integer: "string" ``` If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller will validate it further. #### Comprehensive Validation NGINX Ingress Controller validates the fields of the VirtualServer and VirtualServerRoute resources. If a resource is invalid, NGINX Ingress Controller will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller will ignore it. You can check if NGINX Ingress Controller successfully applied the configuration for a VirtualServer. For our example `cafe` VirtualServer, we can run: ```shell kubectl describe vs cafe ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated ``` Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid resource, NGINX Ingress Controller will reject it and emit a Rejected event. For example, if you create a VirtualServer `cafe` with two upstream with the same name `tea`, you will get: ```shell kubectl describe vs cafe ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 12s nginx-ingress-controller VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" ``` Note how the events section includes a Warning event with the Rejected reason. Additionally, this information is also available in the `status` field of the VirtualServer resource. Note the Status section of the VirtualServer: ```shell kubectl describe vs cafe ``` ```text ... Status: External Endpoints: Ip: 12.13.23.123 Ports: [80,443] Message: VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" Reason: Rejected State: Invalid ``` NGINX Ingress Controller validates VirtualServerRoute resources in a similar way. **Note**: If you make an existing resource invalid, NGINX Ingress Controller will reject it and remove the corresponding configuration from NGINX. ## Multiple regex routes in a VirtualServerRoute NGINX Ingress Controller lets you reference the same VirtualServerRoute from multiple regex routes in a VirtualServer. This means you can group related regex paths by concern — for example, all `/api` paths go to one team's VirtualServerRoute and all `/images` paths go to another. ### How it works When NGINX Ingress Controller processes a VirtualServer, it collects every regex route (`~` or `~*`) that references the same VirtualServerRoute. It then validates the collected VirtualServer paths against the VirtualServerRoute's subroutes as a single set. Each subroute produces a separate NGINX `location` block in the generated configuration. ### Set match requirement The VirtualServer paths and VirtualServerRoute subroutes must form a **bidirectional set match**: - Every VirtualServer regex path that references the VirtualServerRoute must have a corresponding subroute. - Every subroute in the VirtualServerRoute must be referenced by a VirtualServer regex path. If a VirtualServer has a path that the referenced VirtualServerRoute does not, or if a VirtualServerRoute has a subpath that isn't present in the VirtualServer, NGINX Ingress Controller rejects the VirtualServerRoute and the VirtualServer enters a warning state. | Direction | Rule | Error if violated | | --- | --- | --- | | VirtualServer → VirtualServerRoute | Every VirtualServer regex path must appear as a subroute | "subroute with path '...' is missing" | | VirtualServerRoute → VirtualServer | Every subroute must be referenced by a VirtualServer regex path | "subroute path '...' is not referenced by any VS route" | ### Configure multiple regex routes The following example shows a VirtualServer with five routes. Two case-sensitive regex routes delegate to `vsr-api`, and two case-insensitive regex routes delegate to `vsr-media`. The `/health` route is handled directly by the VirtualServer. #### VirtualServer ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: api-gateway spec: host: api-gateway.example.com routes: - path: /health action: return: code: 200 type: text/plain body: "OK" - path: "~/api/v1" route: vsr-api - path: "~/api/v2" route: vsr-api - path: "~*/images/jpg" route: vsr-media - path: "~*/images/png" route: vsr-media ``` #### VirtualServerRoute for API routes ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: vsr-api spec: host: api-gateway.example.com subroutes: - path: "~/api/v1" action: pass: api-v1 - path: "~ /api/v2" action: pass: api-v2 ``` Notice that the second subroute uses `"~ /api/v2"` (with a space) while the VirtualServer declares `"~/api/v2"` (without a space). NGINX Ingress Controller normalizes both to the same value, so the configuration is valid. See [Path normalization](#path-normalization) for details. #### VirtualServerRoute for media routes ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: vsr-media spec: host: api-gateway.example.com subroutes: - path: "~*/images/jpg" action: pass: images-jpg - path: "~*/images/png" action: pass: images-png ``` #### Generated NGINX configuration Each subroute produces a separate `location` block. The generated NGINX configuration for the four regex subroutes looks like this: ```nginx location ~ "/api/v1" { ... } location ~ "/api/v2" { ... } location ~* "/images/jpg" { ... } location ~* "/images/png" { ... } ``` #### Request routing | Request URI | Matching location | VirtualServerRoute | | --- | --- | --- | | `/health` | `/health` (prefix, direct VirtualServer action) | None | | `/api/v1` | `~ "/api/v1"` | `vsr-api` | | `/api/v2` | `~ "/api/v2"` | `vsr-api` | | `/images/jpg` | `~* "/images/jpg"` | `vsr-media` | | `/images/png` | `~* "/images/png"` | `vsr-media` | ### Path normalization NGINX Ingress Controller strips whitespace between the regex modifier (`~` or `~*`) and the path before comparing values. This means `"~/api/v1"` and `"~ /api/v1"` normalize to the same value. Normalization applies to all path comparisons: - **Set match validation**: A VirtualServer path `"~/api/v2"` matches a VirtualServerRoute subroute `"~ /api/v2"` because both normalize to `"~/api/v2"`. - **Duplicate detection**: If a single VirtualServerRoute contains both `"~*/images/jpg"` and `"~* /images/jpg"` as subroutes, they normalize to the same value and NGINX Ingress Controller rejects the VirtualServerRoute with a duplicate path error. **important:** Using `"~ /api/v1"` in a VirtualServerRoute to match `"~/api/v1"` in a VirtualServer is valid — the paths are in different resources and represent the same logical route. Having both `"~/api/v1"` and `"~ /api/v1"` as separate subroutes in the **same** VirtualServerRoute is invalid — they resolve to the same normalized path and are treated as duplicates. ### Validation rules NGINX Ingress Controller enforces the following rules for regex route delegation: - Every VirtualServer regex path referencing a VirtualServerRoute must have a matching subroute in that VirtualServerRoute. - Every subroute in the VirtualServerRoute must be referenced by a VirtualServer regex path. - Subroute paths must be unique after normalization. Paths that differ only in whitespace between the modifier and the URI are treated as duplicates. - All VirtualServer routes referencing the same VirtualServerRoute must use the same modifier category. You can't have a regex route (`~`) and a prefix route (`/`) both referencing the same VirtualServerRoute. - Case-sensitive (`~`) and case-insensitive (`~*`) modifiers can reference the same VirtualServerRoute because both are in the regex category. - Exact match routes (`=`) still require exactly one subroute. Only regex routes support multiple subroutes per VirtualServerRoute. ### Validation errors When NGINX Ingress Controller detects a validation failure, it rejects the VirtualServerRoute and the VirtualServer enters a warning state. Other VirtualServerRoutes and direct VirtualServer routes continue to serve traffic normally. #### Missing subroute This error occurs when the VirtualServer references a regex path through a VirtualServerRoute, but the VirtualServerRoute doesn't have a subroute for that path. For example, the VirtualServer references both `~/api/v1` and `~/api/v2` through `vsr-api`, but the VirtualServerRoute only defines `~/api/v1`: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: vsr-api spec: host: api-gateway.example.com subroutes: - path: "~/api/v1" action: pass: api-v1 ``` NGINX Ingress Controller rejects the VirtualServerRoute with the following error: ```text VirtualServerRoute default/vsr-api is invalid: spec.subroutes: Invalid value: "subroutes": subroute with path '~/api/v2' is missing; all VS route paths must be covered by VSR subroutes ``` To fix this, add the missing subroute (`~/api/v2`) to the VirtualServerRoute, or remove the VirtualServer route that references it. #### Extra subroute This error occurs when the VirtualServerRoute has a subroute that no VirtualServer route references. For example, the VirtualServerRoute defines `~/api/v1`, `~/api/v2`, and `~/api/v3`, but the VirtualServer only references `~/api/v1` and `~/api/v2`: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: vsr-api spec: host: api-gateway.example.com subroutes: - path: "~/api/v1" action: pass: api-v1 - path: "~/api/v2" action: pass: api-v2 - path: "~/api/v3" action: pass: api-v3 ``` NGINX Ingress Controller rejects the VirtualServerRoute with the following error: ```text VirtualServerRoute default/vsr-api is invalid: spec.subroutes[2].path: Invalid value: "~/api/v3": subroute path '~/api/v3' is not referenced by any VS route; all VSR subroutes must be referenced ``` To fix this, remove the extra subroute from the VirtualServerRoute, or add a corresponding regex route in the VirtualServer. #### Duplicate paths after normalization This error occurs when two subroutes in the same VirtualServerRoute resolve to the same path after normalization. Because NGINX Ingress Controller strips whitespace between the modifier and the URI, paths that appear different in YAML can be identical after normalization. For example, the following VirtualServerRoute has `"~/api/v1"` and `"~ /api/v1"` as separate subroutes: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: vsr-api spec: host: api-gateway.example.com subroutes: - path: "~/api/v1" action: pass: api-v1 - path: "~ /api/v1" action: pass: api-v1-alt ``` Both paths normalize to `"~/api/v1"`. NGINX Ingress Controller rejects the VirtualServerRoute: ```text VirtualServerRoute default/vsr-api is invalid: spec.subroutes[0].path: Duplicate value: "~/api/v1", spec.subroutes[1].path: Duplicate value: "~ /api/v1" ``` To fix this, remove the duplicate subroute. If you need to match the same pattern, use a single subroute. ### See also - [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) for the full field reference. - NGINX [location directive documentation](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) for how NGINX evaluates regex locations. - [Multiple regex routes examples](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/vsr-multiple-regex-routes) on GitHub for runnable manifests and additional error case examples. ## Customization using ConfigMap You can customize the NGINX configuration for VirtualServer and VirtualServerRoutes resources using the [ConfigMap](/nic/configuration/global-configuration/configmap-resource.md). Most of the ConfigMap keys are supported, with the following exceptions: - `proxy-hide-headers` - `proxy-pass-headers` - `hsts` - `hsts-max-age` - `hsts-include-subdomains` - `hsts-behind-proxy` - `redirect-to-https` - `ssl-redirect` - `http-redirect-code` -------------------------------------------------------------------------------- title: "VirtualServer and VirtualServerRoute resources" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/virtualserver-and-virtualserverroute-resources/index.md -------------------------------------------------------------------------------- This document is reference material for the VirtualServer and VirtualServerRoute resources used by F5 NGINX Ingress Controller LTS. VirtualServer and VirtualServerRoute resources are load balancing configurations recommended as an alternative to the Ingress resource. They enable use cases not supported with the Ingress resource, such as traffic splitting and advanced content-based routing. The resources are implemented as [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The GitHub repository has [examples of the resources](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources) for specific use cases. --- ## VirtualServer specification The VirtualServer resource defines load balancing configuration for a domain name, such as `example.com`. The example below shows a VirtualServer with both traditional static routes and a dynamic route using `routeSelector`. The route with path `/dynamic` will automatically include any VirtualServerRoute resources that have the label `app: cafe`: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe spec: host: cafe.example.com listener: http: http-8083 https: https-8443 tls: secret: cafe-secret gunzip: on upstreams: - name: tea service: tea-svc port: 80 - name: coffee service: coffee-svc port: 80 routes: - path: /tea action: pass: tea - path: /coffee action: pass: coffee - path: ~ ^/decaf/.*\\.jpg$ action: pass: coffee - path: =/green/tea action: pass: tea - path: /dynamic routeSelector: matchLabels: app: cafe ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. The ``host`` value needs to be unique among all Ingress and VirtualServer resources. See also [Handling Host and Listener Collisions](/nic/lts/configuration/host-and-listener-collisions.md). | ``string`` | Yes | |``listener`` | Sets a custom HTTP and/or HTTPS listener. Valid fields are `listener.http` and `listener.https`. Each field must reference the name of a valid listener defined in a GlobalConfiguration resource | [listener](#virtualserverlistener) | No | |``tls`` | The TLS termination configuration. | [tls](#virtualservertls) | No | |``gunzip`` | Enables or disables [decompression](/nginx/admin-guide/web-server/compression.md) of gzipped responses for clients. Allowed values “on”/“off”, “true”/“false” or “yes”/“no”. If the ``gunzip`` value is not set, it defaults to ``off``. | ``boolean`` | No | |``externalDNS`` | The externalDNS configuration for a VirtualServer. | [externalDNS](#virtualserverexternaldns) | No | |``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer. | ``string`` | No | |``policies`` | A list of policies. | [[]policy](#virtualserverpolicy) | No | |``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | |``routes`` | A list of routes. | [[]route](#virtualserverroute) | No | |``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServer resource. | ``string`` | No | |``internalRoute`` | Specifies if the VirtualServer resource is an internal route or not. | ``boolean`` | No | |``http-snippets`` | Sets a custom snippet in the http context. | ``string`` | No | |``server-snippets`` | Sets a custom snippet in server context. Overrides the ``server-snippets`` ConfigMap key. | ``string`` | No | ### VirtualServer.TLS The tls field defines TLS configuration for a VirtualServer. For example: ```yaml secret: cafe-secret redirect: enable: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``secret`` | The name of a secret with a TLS certificate and key. The secret must belong to the same namespace as the VirtualServer. The secret must be of the type ``kubernetes.io/tls`` and contain keys named ``tls.crt`` and ``tls.key`` that contain the certificate and private key as described [here](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls). If the secret doesn't exist or is invalid, NGINX will break any attempt to establish a TLS connection to the host of the VirtualServer. If the secret is not specified but [wildcard TLS secret](/nic/lts/configuration/global-configuration/command-line-arguments.md#cmdoption-wildcard-tls-secret) is configured, NGINX will use the wildcard secret for TLS termination. | ``string`` | No | |``redirect`` | The redirect configuration of the TLS for a VirtualServer. | [tls.redirect](#virtualservertlsredirect) | No | ### VirtualServer.TLS.Redirect | |``cert-manager`` | The cert-manager configuration of the TLS for a VirtualServer. | [tls.cert-manager](#virtualservertlscertmanager) | No | ### VirtualServer.TLS.CertManager | ### VirtualServer.TLS.Redirect The redirect field configures a TLS redirect for a VirtualServer: ```yaml enable: true code: 301 basedOn: scheme ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables a TLS redirect for a VirtualServer. The default is ``False``. | ``boolean`` | No | |``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | |``basedOn`` | The attribute of a request that NGINX will evaluate to send a redirect. The allowed values are ``scheme`` (the scheme of the request) or ``x-forwarded-proto`` (the ``X-Forwarded-Proto`` header of the request). The default is ``scheme``. | ``string`` | No | ### VirtualServer.Policy | ### VirtualServer.TLS.CertManager The cert-manager field configures x509 automated Certificate management for VirtualServer resources using cert-manager (cert-manager.io). Please see the [cert-manager configuration documentation](https://cert-manager.io/docs/configuration/) for more information on deploying and configuring Issuers. Example: ```yaml cert-manager: cluster-issuer: "my-issuer-name" ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``issuer`` | the name of an Issuer. An Issuer is a cert-manager resource which describes the certificate authority capable of signing certificates. The Issuer must be in the same namespace as the VirtualServer resource. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | |``cluster-issuer`` | the name of a ClusterIssuer. A ClusterIssuer is a cert-manager resource which describes the certificate authority capable of signing certificates. It does not matter which namespace your VirtualServer resides, as ClusterIssuers are non-namespaced resources. Please note that one of `issuer` and `cluster-issuer` are required, but they are mutually exclusive - one and only one must be defined. | ``string`` | No | |``issuer-kind`` | The kind of the external issuer resource, for example AWSPCAIssuer. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | |``issuer-group`` | The API group of the external issuer controller, for example awspca.cert-manager.io. This is only necessary for out-of-tree issuers. This cannot be defined if `cluster-issuer` is also defined. | ``string`` | No | |``common-name`` | This field allows you to configure spec.commonName for the Certificate to be generated. This configuration adds a CN to the x509 certificate. | ``string`` | No | |``duration`` | This field allows you to configure spec.duration field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | |``renew-before`` | this annotation allows you to configure spec.renewBefore field for the Certificate to be generated. Must be specified using a [Go time.Duration](https://pkg.go.dev/time#ParseDuration) string format, which does not allow the d (days) suffix. You must specify these values using s, m, and h suffixes instead. | ``string`` | No | |``usages`` | This field allows you to configure spec.usages field for the Certificate to be generated. Pass a string with comma-separated values i.e. ``key agreement,digital signature, server auth``. An exhaustive list of supported key usages can be found in the [the cert-manager api documentation](https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.KeyUsage). | ``string`` | No | |``issue-temp-cert`` | When ``true``, ask cert-manager for a [temporary self-signed certificate](https://cert-manager.io/docs/usage/certificate/#temporary-certificates-while-issuing) pending the issuance of the Certificate. This allows HTTPS-only servers to use ACME HTTP01 challenges when the TLS secret does not exist yet. | ``boolean`` | No | ### VirtualServer.Listener The listener field defines a custom HTTP and/or HTTPS listener. The respective listeners used must reference the name of a listener defined using a [GlobalConfiguration](/nic/lts/configuration/global-configuration/globalconfiguration-resource.md) resource. For example: ```yaml http: http-8083 https: https-8443 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``http`` | The name of am HTTP listener defined in a [GlobalConfiguration](/nic/lts/configuration/global-configuration/globalconfiguration-resource.md) resource. | ``string`` | No | |``https`` | The name of an HTTPS listener defined in a [GlobalConfiguration](/nic/lts/configuration/global-configuration/globalconfiguration-resource.md) resource. | ``string`` | No | ### VirtualServer.ExternalDNS The externalDNS field configures controlling DNS records dynamically for VirtualServer resources using [ExternalDNS](https://github.com/kubernetes-sigs/external-dns). Please see the [ExternalDNS configuration documentation](https://kubernetes-sigs.github.io/external-dns/) for more information on deploying and configuring ExternalDNS and Providers. Example: ```yaml enable: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables ExternalDNS integration for a VirtualServer resource. The default is ``false``. | ``string`` | No | |``labels`` | Configure labels to be applied to the Endpoint resources that will be consumed by ExternalDNS. | ``map[string]string`` | No | |``providerSpecific`` | Configure provider specific properties which holds the name and value of a configuration which is specific to individual DNS providers. | [[]ProviderSpecific](#virtualserverexternaldnsproviderspecific) | No | |``recordTTL`` | TTL for the DNS record. This defaults to 0 if not defined. See [the ExternalDNS TTL documentation for provider-specific defaults](https://kubernetes-sigs.github.io/external-dns/v0.14.2/ttl/#providers) | ``int64`` | No | |``recordType`` | The record Type that should be created, e.g. "A", "AAAA", "CNAME". This is automatically computed based on the external endpoints if not defined. | ``string`` | No | ### VirtualServer.ExternalDNS.ProviderSpecific The providerSpecific field of the externalDNS block allows the specification of provider specific properties which is a list of key value pairs of configurations which are specific to individual DNS providers. Example: ```yaml - name: my-name value: my-value - name: my-name2 value: my-value2 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the key value pair. | ``string`` | Yes | |``value`` | The value of the key value pair. | ``string`` | Yes | ### VirtualServer.Policy The policy field references a [Policy resource](/nic/lts/configuration/policy-resource/before-you-begin.md) by its name and optional namespace. For example: ```yaml name: access-control ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of a policy. If the policy doesn't exist or invalid, NGINX will respond with an error response with the `500` status code. | ``string`` | Yes | |``namespace`` | The namespace of a policy. If not specified, the namespace of the VirtualServer resource is used. | ``string`` | No | ### VirtualServer.Route The route defines rules for matching client requests to actions like passing a request to an upstream. For example: ```yaml path: /tea action: pass: tea ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``path`` | The path of the route. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix (must start with ``/`` ) or an exact match (must start with ``=`` ), the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all routes of the VirtualServer. Check the [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directive for more information. | ``string`` | Yes | |``policies`` | A list of policies. The policies override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying Policies](/nic/lts/configuration/policy-resource/using-policy.md#attach-policies-to-a-resource) for more details. | [[]policy](#virtualserverpolicy) | No | |``action`` | The default action to perform for a request. | [action](#action) | No | |``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServer route. | ``string`` | No | |``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | |``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | |``route`` | The name of a VirtualServerRoute resource that defines this route. If the VirtualServerRoute belongs to a different namespace than the VirtualServer, you need to include the namespace. For example, ``tea-namespace/tea``. | ``string`` | No | |``routeSelector`` | The RouteSelector allows selecting VirtualServerRoute resources using label selectors instead of specifying them by name and namespace. | [routeSelector](#virtualserverrouterouteselector) | No | |``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | |``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` ConfigMap key. | ``string`` | No | \* -- a route must include exactly one of the following: `action`, `splits`, `route` or `routeSelector` ### VirtualServer.Route.RouteSelector The RouteSelector field allows you to dynamically select VirtualServerRoute resources using label selectors instead of specifying them by name and namespace. This provides a more flexible and scalable way to associate VirtualServerRoutes with a VirtualServer route. See the [VirtualServerRoute specification](#virtualserverroute-specification) section below for examples of how to use RouteSelector with VirtualServerRoute resources. |Field | Description | Type | Required | | ---| ---| ---| --- | |``matchLabels`` | A map of key-value pairs. Each key-value pair in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. | ``map[string]string`` | Yes | |``matchExpressions`` | A list of label selector requirements. The requirements are ANDed. For more information on label selector requirements, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). | N/A | No | ## VirtualServerRoute specification The VirtualServerRoute resource defines a route for a VirtualServer. It can consist of one or multiple subroutes. The VirtualServerRoute is an alternative to [Mergeable Ingress types](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md). VirtualServer routes can reference VirtualServerRoute resources in two ways: by name using the `route` field, or dynamically using the `routeSelector` field with label selectors. The `routeSelector` approach allows you to add new VirtualServerRoute resources without modifying the VirtualServer configuration. #### Standard In this example, the VirtualServer `cafe` from the namespace `cafe-ns` defines a route that references a specific VirtualServerRoute by name. VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe-ns spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 routes: - path: /tea action: pass: tea - path: /coffee route: coffee-ns/coffee ``` VirtualServerRoute: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: coffee namespace: coffee-ns spec: host: cafe.example.com upstreams: - name: latte service: latte-svc port: 80 - name: espresso service: espresso-svc port: 80 subroutes: - path: /coffee/latte action: pass: latte - path: /coffee/espresso action: pass: espresso ``` #### RouteSelector In this example, the VirtualServer `cafe` from the namespace `cafe-ns` uses `routeSelector` to dynamically select any VirtualServerRoute with the label `app: coffee`. VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe-ns spec: host: cafe.example.com upstreams: - name: tea service: tea-svc port: 80 routes: - path: /tea action: pass: tea - path: /decaf routeSelector: matchLabels: app: coffee ``` VirtualServerRoute: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: decaf namespace: coffee-ns labels: app: coffee spec: host: cafe.example.com upstreams: - name: latte service: latte-svc port: 80 - name: espresso service: espresso-svc port: 80 subroutes: - path: /decaf/latte action: pass: latte - path: /decaf/espresso action: pass: espresso ``` Note that each subroute must have a `path` that starts with the same prefix (here `/coffee`), which is defined in the route of the VirtualServer. Additionally, the `host` in the VirtualServerRoute must be the same as the `host` of the VirtualServer. |Field | Description | Type | Required | | ---| ---| ---| --- | |``host`` | The host (domain name) of the server. Must be a valid subdomain as defined in RFC 1123, such as ``my-app`` or ``hello.example.com``. When using a wildcard domain like ``*.example.com`` the domain must be contained in double quotes. Must be the same as the ``host`` of the VirtualServer that references this resource. | ``string`` | Yes | |``upstreams`` | A list of upstreams. | [[]upstream](#upstream) | No | |``subroutes`` | A list of subroutes. | [[]subroute](#virtualserverroutesubroute) | No | |``ingressClassName`` | Specifies which Ingress Controller must handle the VirtualServerRoute resource. Must be the same as the ``ingressClassName`` of the VirtualServer that references this resource. | ``string``_ | No | ### VirtualServerRoute.Subroute The subroute defines rules for matching client requests to actions like passing a request to an upstream. For example: ```yaml path: /coffee action: pass: coffee ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``path`` | The path of the subroute. NGINX will match it against the URI of a request. Possible values are: a prefix ( ``/`` , ``/path`` ), an exact match ( ``=/exact/match`` ), a case insensitive regular expression ( ``~*^/Bar.*\.jpg`` ) or a case sensitive regular expression ( ``~^/foo.*\.jpg`` ). In the case of a prefix, the path must start with the same path as the path of the route of the VirtualServer that references this resource. In the case of an exact or regex match, the path must be the same as the path of the route of the VirtualServer that references this resource. A matching path of the route of the VirtualServer but in different type is not accepted, e.g. a regex path (`~/match`) cannot be used with a prefix path in VirtualServer (`/match`) In the case of a prefix or an exact match, the path must not include any whitespace characters, ``{`` , ``}`` or ``;``. In the case of the regex matches, all double quotes ``"`` must be escaped and the match can't end in an unescaped backslash ``\``. The path must be unique among the paths of all subroutes of the VirtualServerRoute. | ``string`` | Yes | |``policies`` | A list of policies. The policies override *all* policies defined in the route of the VirtualServer that references this resource. The policies also override the policies of the same type defined in the ``spec`` of the VirtualServer. See [Applying Policies](/nic/lts/configuration/policy-resource/using-policy.md#attach-policies-to-a-resource) for more details. | [[]policy](#virtualserverpolicy) | No | |``action`` | The default action to perform for a request. | [action](#action) | No | |``dos`` | A reference to a DosProtectedResource, setting this enables DOS protection of the VirtualServerRoute subroute. | ``string`` | No | |``splits`` | The default splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | |``matches`` | The matching rules for advanced content-based routing. Requires the default ``action`` or ``splits``. Unmatched requests will be handled by the default ``action`` or ``splits``. | [matches](#match) | No | |``errorPages`` | The custom responses for error codes. NGINX will use those responses instead of returning the error responses from the upstream servers or the default responses generated by NGINX. A custom response can be a redirect or a canned response. For example, a redirect to another URL if an upstream server responded with a 404 status code. | [[]errorPage](#errorpage) | No | |``location-snippets`` | Sets a custom snippet in the location context. Overrides the ``location-snippets`` of the VirtualServer (if set) or the ``location-snippets`` ConfigMap key. | ``string`` | No | \* -- a subroute must include exactly one of the following: `action` or `splits`. ## Common VirtualServer and VirtualServerRoute specifications ### Upstream The upstream defines a destination for the routing configuration. For example: ```yaml name: tea service: tea-svc subselector: version: canary port: 80 lb-method: round_robin fail-timeout: 10s max-fails: 1 max-conns: 32 keepalive: 32 connect-timeout: 30s read-timeout: 30s send-timeout: 30s next-upstream: "error timeout non_idempotent" next-upstream-timeout: 5s next-upstream-tries: 10 client-max-body-size: 2m tls: enable: true ``` **Note**: The WebSocket protocol is supported without any additional configuration. |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes | |``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). If the Service belongs to a different namespace than the VirtualServer or VirtualServerRoute, you need to include the namespace. For example, ``tea-namespace/tea``. If the service doesn't exist, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. Services of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) are also supported in the same namespace (check the [prerequisites](https://github.com/nginx/kubernetes-ingress/tree/v/examples/ingress-resources/externalname-services#prerequisites) ). | ``string`` | Yes | |``subselector`` | Selects the pods within the service using label keys and values. By default, all pods of the service are selected. Note: the specified labels are expected to be present in the pods when they are created. If the pod labels are updated, NGINX Ingress Controller LTS will not see that change until the number of the pods is changed. | ``map[string]string`` | No | |``use-cluster-ip`` | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as NGINX Ingress Controller LTS will configure NGINX with only one upstream server that will match the service Cluster IP. | ``boolean`` | No | |``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. The port must fall into the range ``1..65535``. | ``uint16`` | Yes | |``lb-method`` | The load [balancing method](/nginx/admin-guide/load-balancer/http-load-balancer.md#choosing-a-load-balancing-method). To use the round-robin method, specify ``round_robin``. The default is specified in the ``lb-method`` ConfigMap key. | ``string`` | No | |``fail-timeout`` | The time during which the specified number of unsuccessful attempts to communicate with an upstream server should happen to consider the server unavailable. See the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the server directive. The default is set in the ``fail-timeout`` ConfigMap key. | ``string`` | No | |``max-fails`` | The number of unsuccessful attempts to communicate with an upstream server that should happen in the duration set by the ``fail-timeout`` to consider the server unavailable. See the [max_fails](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_fails) parameter of the server directive. The default is set in the ``max-fails`` ConfigMap key. | ``int`` | No | |``max-conns`` | The maximum number of simultaneous active connections to an upstream server. See the [max_conns](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#max_conns) parameter of the server directive. By default there is no limit. Note: if keepalive connections are enabled, the total number of active and idle keepalive connections to an upstream server may exceed the ``max_conns`` value. | ``int`` | No | |``keepalive`` | Configures the cache for connections to upstream servers. The value ``0`` disables the cache. See the [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) directive. The default is set in the ``keepalive`` ConfigMap key. | ``int`` | No | |``connect-timeout`` | The timeout for establishing a connection with an upstream server. See the [proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) directive. The default is specified in the ``proxy-connect-timeout`` ConfigMap key. | ``string`` | No | |``read-timeout`` | The timeout for reading a response from an upstream server. See the [proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) directive. The default is specified in the ``proxy-read-timeout`` ConfigMap key. | ``string`` | No | |``send-timeout`` | The timeout for transmitting a request to an upstream server. See the [proxy_send_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) directive. The default is specified in the ``proxy-send-timeout`` ConfigMap key. | ``string`` | No | |``next-upstream`` | Specifies in which cases a request should be passed to the next upstream server. See the [proxy_next_upstream](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) directive. The default is ``error timeout``. | ``string`` | No | |``next-upstream-timeout`` | The time during which a request can be passed to the next upstream server. See the [proxy_next_upstream_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. The ``0`` value turns off the time limit. The default is ``0``. | ``string`` | No | |``next-upstream-tries`` | The number of possible tries for passing a request to the next upstream server. See the [proxy_next_upstream_tries](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. The ``0`` value turns off this limit. The default is ``0``. | ``int`` | No | |``client-max-body-size`` | Sets the maximum allowed size of the client request body. See the [client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) directive. The default is set in the ``client-max-body-size`` ConfigMap key. | ``string`` | No | |``client-body-buffer-size`` | Sets the value of the [client_body_buffer_size](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) directive. The default is set in the ``client-body-buffer-size`` ConfigMap key. | ``string`` | No | |``tls`` | The TLS configuration for the Upstream. | [tls](#upstreamtls) | No | |``healthCheck`` | The health check configuration for the Upstream. See the [health_check](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive. | [healthcheck](#upstreamhealthcheck) | No | |``slow-start`` | The slow start allows an upstream server to gradually recover its weight from 0 to its nominal value after it has been recovered or became available or when the server becomes available after a period of time it was considered unavailable. By default, the slow start is disabled. See the [slow_start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start) parameter of the server directive. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods and will be ignored. | ``string`` | No | |``queue`` | Configures a queue for an upstream. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request. By default, no queue is configured. | [queue](#upstreamqueue) | No | |``buffering`` | Enables buffering of responses from the upstream server. See the [proxy_buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) directive. The default is set in the ``proxy-buffering`` ConfigMap key. | ``boolean`` | No | |``buffers`` | Configures the buffers used for reading a response from the upstream server for a single connection. | [buffers](#upstreambuffers) | No | |``buffer-size`` | Sets the size of the buffer used for reading the first part of a response received from the upstream server. See the [proxy_buffer_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) directive. The default is set in the ``proxy-buffer-size`` ConfigMap key. | ``string`` | No | |``busy-buffers-size`` | Sets the size of the buffer used for reading a response from the upstream server when the response is larger than the ``buffer-size``. See the [proxy_busy_buffers_size](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) directive. The default is set in the ``proxy-busy-buffers-size`` ConfigMap key. | ``string`` | No | |``ntlm`` | Allows proxying requests with NTLM Authentication. See the [ntlm](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#ntlm) directive. In order for NTLM authentication to work, it is necessary to enable keepalive connections to upstream servers using the ``keepalive`` field. | ``boolean`` | No | |``type`` |The type of the upstream. Supported values are ``http`` and ``grpc``. The default is ``http``. For gRPC, it is necessary to enable HTTP/2 in the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource.md#listeners) and configure TLS termination in the VirtualServer. | ``string`` | No | |``backup`` | The name of the backup service of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname). This will be used when the primary servers are unavailable. Note: The parameter cannot be used along with the ``random`` , ``hash`` or ``ip_hash`` load balancing methods. | ``string`` | No | |``backupPort`` | The port of the backup service. The backup port is required if the backup service name is provided. The port must fall into the range ``1..65535``. | ``uint16`` | No | ### Upstream.Buffers The buffers field configures the buffers used for reading a response from the upstream server for a single connection: ```yaml number: 4 size: 8K ``` See the [proxy_buffers](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) directive for additional information. |Field | Description | Type | Required | | ---| ---| ---| --- | |``number`` | Configures the number of buffers. The default is set in the ``proxy-buffers`` ConfigMap key. | ``int`` | Yes | |``size`` | Configures the size of a buffer. The default is set in the ``proxy-buffers`` ConfigMap key. | ``string`` | Yes | ### Upstream.TLS |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables HTTPS for requests to upstream servers. The default is ``False`` , meaning that HTTP will be used. Note: by default, NGINX will not verify the upstream server certificate. To enable the verification, configure an [EgressMTLS Policy](/nic/lts/configuration/policy-resource/policy-specification.md#egressmtls). | ``boolean`` | No | ### Upstream.Queue The queue field configures a queue. A client request will be placed into the queue if an upstream server cannot be selected immediately while processing the request: ```yaml size: 10 timeout: 60s ``` See [`queue`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#queue) directive for additional information. |Field | Description | Type | Required | | ---| ---| ---| --- | |``size`` | The size of the queue. | ``int`` | Yes | |``timeout`` | The timeout of the queue. A request cannot be queued for a period longer than the timeout. The default is ``60s``. | ``string`` | No | ### Upstream.Healthcheck The Healthcheck defines an [active health check](/nginx/admin-guide/load-balancer.md#http-health-check). In the example below we enable a health check for an upstream and configure all the available parameters, including the `slow-start` parameter combined with [`mandatory` and `persistent`](/nginx/admin-guide/load-balancer/http-health-check.md#mandatory-health-checks): ```yaml name: tea service: tea-svc port: 80 slow-start: 30s healthCheck: enable: true path: /healthz interval: 20s jitter: 3s fails: 5 passes: 5 port: 8080 tls: enable: true connect-timeout: 10s read-timeout: 10s send-timeout: 10s headers: - name: Host value: my.service statusMatch: "! 500" mandatory: true persistent: true keepalive-time: 60s ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables a health check for an upstream server. The default is ``false``. | ``boolean`` | No | |``path`` | The path used for health check requests. The default is ``/``. This not configurable for gRPC type upstreams. | ``string`` | No | |``interval`` | The interval between two consecutive health checks. The default is ``5s``. | ``string`` | No | |``jitter`` | The time within which each health check will be randomly delayed. By default, there is no delay. | ``string`` | No | |``fails`` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is ``1``. | ``integer`` | No | |``passes`` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is ``1``. | ``integer`` | No | |``port`` | The port used for health check requests. By default, the [server port is used](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check_port). Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | ``integer`` | No | |``tls`` | The TLS configuration used for health check requests. By default, the ``tls`` field of the upstream is used. | [upstream.tls](#upstreamtls) | No | |``connect-timeout`` | The timeout for establishing a connection with an upstream server. By default, the ``connect-timeout`` of the upstream is used. | ``string`` | No | |``read-timeout`` | The timeout for reading a response from an upstream server. By default, the ``read-timeout`` of the upstream is used. | ``string`` | No | |``send-timeout`` | The timeout for transmitting a request to an upstream server. By default, the ``send-timeout`` of the upstream is used. | ``string`` | No | |``headers`` | The request headers used for health check requests. NGINX Plus always sets the ``Host`` , ``User-Agent`` and ``Connection`` headers for health check requests. | [[]header](#header) | No | |``statusMatch`` | The expected response status codes of a health check. By default, the response should have status code 2xx or 3xx. Examples: ``"200"`` , ``"! 500"`` , ``"301-303 307"``. See the documentation of the [match](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html?#match) directive. This not supported for gRPC type upstreams. | ``string`` | No | |``grpcStatus`` | The expected [gRPC status code](https://github.com/grpc/grpc/blob/master/doc/statuscodes.md#status-codes-and-their-use-in-grpc) of the upstream server response to the [Check method](https://github.com/grpc/grpc/blob/master/doc/health-checking.md). Configure this field only if your gRPC services do not implement the gRPC health checking protocol. For example, configure ``12`` if the upstream server responds with `12 (UNIMPLEMENTED)` status code. Only valid on gRPC type upstreams. | ``int`` | No | |``grpcService`` | The gRPC service to be monitored on the upstream server. Only valid on gRPC type upstreams. | ``string`` | No | |``mandatory`` | Require every newly added server to pass all configured health checks before NGINX Plus sends traffic to it. If this is not specified, or is set to false, the server will be initially considered healthy. When combined with [slow-start](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#slow_start), it gives a new server more time to connect to databases and “warm up” before being asked to handle their full share of traffic. | ``bool`` | No | |``persistent`` | Set the initial “up” state for a server after reload if the server was considered healthy before reload. Enabling persistent requires that the mandatory parameter is also set to `true`. | ``bool`` | No | |``keepalive-time`` | Enables [keepalive](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive) connections for health checks and specifies the time during which requests can be processed through one keepalive connection. The default is ``60s``. | ``string`` | No | ### Upstream.SessionCookie The SessionCookie field configures session persistence which allows requests from the same client to be passed to the same upstream server. The information about the designated upstream server is passed in a session cookie generated by NGINX. In the example below, we configure session persistence with a session cookie for an upstream and configure all the available parameters: ```yaml name: tea service: tea-svc port: 80 sessionCookie: enable: true name: srv_id path: / expires: 1h domain: .example.com httpOnly: false secure: true samesite: strict ``` See the [`sticky`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html?#sticky) directive for additional information. The session cookie corresponds to the `sticky cookie` method. |Field | Description | Type | Required | | ---| ---| ---| --- | |``enable`` | Enables session persistence with a session cookie for an upstream server. The default is ``false``. | ``boolean`` | No | |``name`` | The name of the cookie. | ``string`` | Yes | |``path`` | The path for which the cookie is set. | ``string`` | No | |``expires`` | The time for which a browser should keep the cookie. Can be set to the special value ``max`` , which will cause the cookie to expire on ``31 Dec 2037 23:55:55 GMT``. | ``string`` | No | |``domain`` | The domain for which the cookie is set. | ``string`` | No | |``httpOnly`` | Adds the ``HttpOnly`` attribute to the cookie. | ``boolean`` | No | |``secure`` | Adds the ``Secure`` attribute to the cookie. | ``boolean`` | No | |``samesite`` | Adds the ``SameSite`` attribute to the cookie. The allowed values are: ``strict``, ``lax``, ``none`` | ``string`` | No | ### Action The action defines an action to perform for a request. In the example below, client requests are passed to an upstream `coffee`: ```yaml path: /coffee action: pass: coffee ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``pass`` | Passes requests to an upstream. The upstream with that name must be defined in the resource. | ``string`` | No | |``redirect`` | Redirects requests to a provided URL. | [action.redirect](#actionredirect) | No | |``return`` | Returns a preconfigured response. | [action.return](#actionreturn) | No | |``proxy`` | Passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). | [action.proxy](#actionproxy) | No | \* -- an action must include exactly one of the following: `pass`, `redirect`, `return` or `proxy`. ### Action.Redirect The redirect action defines a redirect to return for a request. In the example below, client requests are passed to a url `http://www.nginx.com`: ```yaml redirect: url: http://www.nginx.com code: 301 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` , ``$http_x_forwarded_proto`` , ``$request_uri`` , ``$host``. Variables must be enclosed in curly braces. For example: ``${host}${request_uri}``. | ``string`` | Yes | |``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | ### Action.Return The return action defines a preconfigured response for a request. In the example below, NGINX will respond with the preconfigured response for every request: ```yaml return: code: 200 type: text/plain body: "Hello World\n" headers: - name: x-coffee value: espresso ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``code`` | The status code of the response. The allowed values are: ``2XX``, ``4XX`` or ``5XX``. The default is ``200``. | ``int`` | No | |``type`` | The MIME type of the response. The default is ``text/plain``. | ``string`` | No | |``body`` | The body of the response. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``Request is ${request_uri}\n``. | ``string`` | Yes | |``headers`` | The custom headers of the response. | [[]Action.Return.Header](#actionreturnheader) | No | \* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing` and `$connections_waiting`. ### Action.Return.Header The header defines an HTTP Header for a canned response in an actionReturn: ```yaml name: x-coffee value: espresso ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. | ``string`` | Yes | ### Action.Proxy The proxy action passes requests to an upstream with the ability to modify the request/response (for example, rewrite the URI or modify the headers). In the example below, the request URI is rewritten to `/`, and the request and the response headers are modified: ```yaml proxy: upstream: coffee requestHeaders: pass: true set: - name: My-Header value: Value - name: Client-Cert value: ${ssl_client_escaped_cert} responseHeaders: add: - name: My-Header value: Value - name: IC-Nginx-Version value: ${nginx_version} always: true hide: - x-internal-version ignore: - Expires - Set-Cookie pass: - Server rewritePath: / ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``upstream`` | The name of the upstream which the requests will be proxied to. The upstream with that name must be defined in the resource. | ``string`` | Yes | |``requestHeaders`` | The request headers modifications. | [action.Proxy.RequestHeaders](#actionproxyrequestheaders) | No | |``responseHeaders`` | The response headers modifications. | [action.Proxy.ResponseHeaders](#actionproxyresponseheaders) | No | |``rewritePath`` | The rewritten URI. If the route path is a regular expression -- starts with `~` -- the `rewritePath` can include capture groups with ``$1-9``. For example `$1` for the first group, and so on. For more information, check the [rewrite](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/rewrites) example. | ``string`` | No | ### Action.Proxy.RequestHeaders The RequestHeaders field modifies the headers of the request to the proxied upstream server. |Field | Description | Type | Required | | ---| ---| ---| --- | |``pass`` | Passes the original request headers to the proxied upstream server. See the [proxy_pass_request_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers) directive for more information. Default is true. | ``bool`` | No | |``set`` | Allows redefining or appending fields to present request headers passed to the proxied upstream servers. See the [proxy_set_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header) directive for more information. | [[]header](#actionproxyrequestheaderssetheader) | No | ### Action.Proxy.RequestHeaders.Set.Header The header defines an HTTP Header: ```yaml name: My-Header value: My-Value ``` It is possible to override the default value of the `Host` header, which NGINX Ingress Controller LTS sets to [`$host`](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_host): ```yaml name: Host value: example.com ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | \* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` and `$jwt_header_`. ### Action.Proxy.ResponseHeaders The ResponseHeaders field modifies the headers of the response to the client. |Field | Description | Type | Required | | ---| ---| ---| --- | |``hide`` | The headers that will not be passed* in the response to the client from a proxied upstream server. See the [proxy_hide_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directive for more information. | ``[]string`` | No | |``pass`` | Allows passing the hidden header fields* to the client from a proxied upstream server. See the [proxy_pass_header](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_header) directive for more information. | ``[]string`` | No | |``ignore`` | Disables processing of certain headers** to the client from a proxied upstream server. See the [proxy_ignore_headers](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers) directive for more information. | ``[]string`` | No | |``add`` | Adds headers to the response to the client. | [[]addHeader](#addheader) | No | \* -- Default hidden headers are: `Date`, `Server`, `X-Pad` and `X-Accel-...`. \** -- The following fields can be ignored: `X-Accel-Redirect`, `X-Accel-Expires`, `X-Accel-Limit-Rate`, `X-Accel-Buffering`, `X-Accel-Charset`, `Expires`, `Cache-Control`, `Set-Cookie` and `Vary`. ### AddHeader The addHeader defines an HTTP Header with an optional `always` field: ```yaml name: My-Header value: My-Value always: true ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. Supports NGINX variables*. Variables must be enclosed in curly brackets. For example: ``${scheme}``. | ``string`` | No | |``always`` | If set to true, add the header regardless of the response status code**. Default is false. See the [add_header](http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header) directive for more information. | ``bool`` | No | \* -- Supported NGINX variables: `$request_uri`, `$request_method`, `$request_body`, `$scheme`, `$http_`, `$args`, `$arg_`, `$cookie_`, `$host`, `$request_time`, `$request_length`, `$nginx_version`, `$pid`, `$connection`, `$remote_addr`, `$remote_port`, `$time_iso8601`, `$time_local`, `$server_addr`, `$server_port`, `$server_name`, `$server_protocol`, `$connections_active`, `$connections_reading`, `$connections_writing`, `$connections_waiting`, `$ssl_cipher`, `$ssl_ciphers`, `$ssl_client_cert`, `$ssl_client_escaped_cert`, `$ssl_client_fingerprint`, `$ssl_client_i_dn`, `$ssl_client_i_dn_legacy`, `$ssl_client_raw_cert`, `$ssl_client_s_dn`, `$ssl_client_s_dn_legacy`, `$ssl_client_serial`, `$ssl_client_v_end`, `$ssl_client_v_remain`, `$ssl_client_v_start`, `$ssl_client_verify`, `$ssl_curves`, `$ssl_early_data`, `$ssl_protocol`, `$ssl_server_name`, `$ssl_session_id`, `$ssl_session_reused`, `$jwt_claim_` and `$jwt_header_`. **note:** If `always` is false, the response header is added only if the response status code is any of `200`, `201`, `204`, `206`, `301`, `302`, `303`, `304`, `307` or `308`. ### Split The split defines a weight for an action as part of the splits configuration. In the example below NGINX passes 80% of requests to the upstream `coffee-v1` and the remaining 20% to `coffee-v2`: ```yaml splits: - weight: 80 action: pass: coffee-v1 - weight: 20 action: pass: coffee-v2 ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``weight`` | The weight of an action. Must fall into the range ``0..100``. The sum of the weights of all splits must be equal to ``100``. | ``int`` | Yes | |``action`` | The action to perform for a request. | [action](#action) | Yes | ### Match The match defines a match between conditions and an action or splits. In the example below, NGINX routes requests with the path `/coffee` to different upstreams based on the value of the cookie `user`: - `user=john` -> `coffee-future` - `user=bob` -> `coffee-deprecated` - If the cookie is not set or not equal to either `john` or `bob`, NGINX routes to `coffee-stable` ```yaml path: /coffee matches: - conditions: - cookie: user value: john action: pass: coffee-future - conditions: - cookie: user value: bob action: pass: coffee-deprecated action: pass: coffee-stable ``` In the next example, NGINX routes requests based on the value of the built-in [`$request_method` variable](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_method), which represents the HTTP method of a request: - all POST requests -> `coffee-post` - all non-POST requests -> `coffee` ```yaml path: /coffee matches: - conditions: - variable: $request_method value: POST action: pass: coffee-post action: pass: coffee ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``conditions`` | A list of conditions. Must include at least 1 condition. | [[]condition](#condition) | Yes | |``action`` | The action to perform for a request. | [action](#action) | No | |``splits`` | The splits configuration for traffic splitting. Must include at least 2 splits. | [[]split](#split) | No | **note:** A match must include exactly one of the following: `action` or `splits`. ### Condition The condition defines a condition in a match. |Field | Description | Type | Required | | ---| ---| ---| --- | |``header`` | The name of a header. Must consist of alphanumeric characters or ``-``. | ``string`` | No | |``cookie`` | The name of a cookie. Must consist of alphanumeric characters or ``_``. | ``string`` | No | |``argument`` | The name of an argument. Must consist of alphanumeric characters or ``_``. | ``string`` | No | |``variable`` | The name of an NGINX variable. Must start with ``$``. See the list of the supported variables below the table. | ``string`` | No | |``value`` | The value to match the condition against. How to define a value is shown below the table. | ``string`` | Yes | **note:** a condition must include exactly one of the following: `header`, `cookie`, `argument` or `variable`. Supported NGINX variables: `$args`, `$http2`, `$https`, `$remote_addr`, `$remote_port`, `$query_string`, `$request`, `$request_body`, `$request_uri`, `$request_method`, `$scheme`. Find the documentation for each variable [here](https://nginx.org/en/docs/varindex.html). The value supports two kinds of matching: - *Case-insensitive string comparison*. For example: - `john` -- case-insensitive matching that succeeds for strings, such as `john`, `John`, `JOHN`. - `!john` -- negation of the case-insensitive matching for john that succeeds for strings, such as `bob`, `anything`, `''` (empty string). - *Matching with a regular expression*. Note that NGINX supports regular expressions compatible with those used by the Perl programming language (PCRE). For example: - `~^yes` -- a case-sensitive regular expression that matches any string that starts with `yes`. For example: `yes`, `yes123`. - `!~^yes` -- negation of the previous regular expression that succeeds for strings like `YES`, `Yes123`, `noyes`. (The negation mechanism is not part of the PCRE syntax). - `~*no$` -- a case-insensitive regular expression that matches any string that ends with `no`. For example: `no`, `123no`, `123NO`. **note:** A value must not include any unescaped double quotes (`"`) and must not end with an unescaped backslash (`\`). For example, the following are invalid values: `some"value`, `somevalue\`. ### ErrorPage The errorPage defines a custom response for a route for the case when either an upstream server responds with (or NGINX generates) an error status code. The custom response can be a redirect or a canned response. See the [error_page](https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page) directive for more information. ```yaml path: /coffee errorPages: - codes: [502, 503] redirect: code: 301 url: https://nginx.org - codes: [404] return: code: 200 body: "Original resource not found, but success!" ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``codes`` | A list of error status codes. | ``[]int`` | Yes | |``redirect`` | The redirect action for the given status codes. | [errorPage.Redirect](#errorpageredirect) | No | |``return`` | The canned response action for the given status codes. | [errorPage.Return](#errorpagereturn) | No | **note:** An errorPage must include exactly one of the following: `return` or `redirect`. ### ErrorPage.Redirect The redirect defines a redirect for an errorPage. In the example below, NGINX responds with a redirect when a response from an upstream server has a 404 status code. ```yaml codes: [404] redirect: code: 301 url: ${scheme}://cafe.example.com/error.html ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``code`` | The status code of a redirect. The allowed values are: ``301`` , ``302`` , ``307`` , ``308``. The default is ``301``. | ``int`` | No | |``url`` | The URL to redirect the request to. Supported NGINX variables: ``$scheme`` and ``$http_x_forwarded_proto``. Variables must be enclosed in curly braces. For example: ``${scheme}``. | ``string`` | Yes | ### ErrorPage.Return The return defines a canned response for an errorPage. In the example below, NGINX responds with a canned response when a response from an upstream server has either 401 or 403 status code. ```yaml codes: [401, 403] return: code: 200 type: application/json body: | {\"msg\": \"You don't have permission to do this\"} headers: - name: x-debug-original-statuses value: ${upstream_status} ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``code`` | The status code of the response. The default is the status code of the original response. | ``int`` | No | |``type`` | The MIME type of the response. The default is ``text/html``. | ``string`` | No | |``body`` | The body of the response. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | Yes | |``headers`` | The custom headers of the response. | [[]errorPage.Return.Header](#errorpagereturnheader) | No | ### ErrorPage.Return.Header The header defines an HTTP Header for a canned response in an errorPage: ```yaml name: x-debug-original-statuses value: ${upstream_status} ``` |Field | Description | Type | Required | | ---| ---| ---| --- | |``name`` | The name of the header. | ``string`` | Yes | |``value`` | The value of the header. Supported NGINX variable: ``$upstream_status`` . Variables must be enclosed in curly braces. For example: ``${upstream_status}``. | ``string`` | No | ## Using VirtualServer and VirtualServerRoute You can use the usual `kubectl` commands to work with VirtualServer and VirtualServerRoute resources, similar to Ingress resources. For example, the following command creates a VirtualServer resource defined in `cafe-virtual-server.yaml` with the name `cafe`: ```shell kubectl apply -f cafe-virtual-server.yaml ``` ```text virtualserver.k8s.nginx.org "cafe" created ``` You can get the resource by running: ```shell kubectl get virtualserver cafe ``` ```text NAME STATE HOST IP PORTS AGE cafe Valid cafe.example.com 12.13.23.123 [80,443] 3m ``` In `kubectl get` and similar commands, you can use the short name `vs` instead of `virtualserver`. Similarly, for VirtualServerRoute you can use `virtualserverroute` or the short name `vsr`. ### Using Snippets Snippets allow you to insert raw NGINX config into different contexts of NGINX configuration. In the example below, we use snippets to configure several NGINX features in a VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe namespace: cafe spec: http-snippets: | limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s; proxy_cache_path /tmp keys_zone=one:10m; host: cafe.example.com tls: secret: cafe-secret server-snippets: | limit_req zone=mylimit burst=20; upstreams: - name: tea service: tea-svc port: 80 - name: coffee service: coffee-svc port: 80 routes: - path: /tea location-snippets: | proxy_cache one; proxy_cache_valid 200 10m; action: pass: tea - path: /coffee action: pass: coffee ``` For additional information, view the [Advanced configuration with Snippets](/nic/lts/configuration/ingress-resources/advanced-configuration-with-snippets.md) topic. ### Validation Two types of validation are available for VirtualServer and VirtualServerRoute resources: - *Structural validation* by the `kubectl` and Kubernetes API server. - *Comprehensive validation* by NGINX Ingress Controller LTS. #### Structural Validation The custom resource definitions for VirtualServer and VirtualServerRoute include structural OpenAPI schema which describes the type of every field of those resources. If you try to create (or update) a resource that violates the structural schema (for example, you use a string value for the port field of an upstream), `kubectl` and Kubernetes API server will reject such a resource: - Example of `kubectl` validation: ```shell kubectl apply -f cafe-virtual-server.yaml ``` ```text error: error validating "cafe-virtual-server.yaml": error validating data: ValidationError(VirtualServer.spec.upstreams[0].port): invalid type for org.nginx.k8s.v1.VirtualServer.spec.upstreams.port: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false ``` - Example of Kubernetes API server validation: ```shell kubectl apply -f cafe-virtual-server.yaml --validate=false ``` ```text The VirtualServer "cafe" is invalid: []: Invalid value: map[string]interface {}{ ... }: validation failure list: spec.upstreams.port in body must be of type integer: "string" ``` If a resource is not rejected (it doesn't violate the structural schema), NGINX Ingress Controller LTS will validate it further. #### Comprehensive Validation NGINX Ingress Controller LTS validates the fields of the VirtualServer and VirtualServerRoute resources. If a resource is invalid, NGINX Ingress Controller LTS will reject it: the resource will continue to exist in the cluster, but NGINX Ingress Controller LTS will ignore it. You can check if NGINX Ingress Controller LTS successfully applied the configuration for a VirtualServer. For our example `cafe` VirtualServer, we can run: ```shell kubectl describe vs cafe ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 16s nginx-ingress-controller Configuration for default/cafe was added or updated ``` Note how the events section includes a Normal event with the AddedOrUpdated reason that informs us that the configuration was successfully applied. If you create an invalid resource, NGINX Ingress Controller LTS will reject it and emit a Rejected event. For example, if you create a VirtualServer `cafe` with two upstream with the same name `tea`, you will get: ```shell kubectl describe vs cafe ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 12s nginx-ingress-controller VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" ``` Note how the events section includes a Warning event with the Rejected reason. Additionally, this information is also available in the `status` field of the VirtualServer resource. Note the Status section of the VirtualServer: ```shell kubectl describe vs cafe ``` ```text ... Status: External Endpoints: Ip: 12.13.23.123 Ports: [80,443] Message: VirtualServer default/cafe is invalid and was rejected: spec.upstreams[1].name: Duplicate value: "tea" Reason: Rejected State: Invalid ``` NGINX Ingress Controller LTS validates VirtualServerRoute resources in a similar way. **Note**: If you make an existing resource invalid, NGINX Ingress Controller LTS will reject it and remove the corresponding configuration from NGINX. ## Customization using ConfigMap You can customize the NGINX configuration for VirtualServer and VirtualServerRoutes resources using the [ConfigMap](/nic/lts/configuration/global-configuration/configmap-resource.md). Most of the ConfigMap keys are supported, with the following exceptions: - `proxy-hide-headers` - `proxy-pass-headers` - `hsts` - `hsts-max-age` - `hsts-include-subdomains` - `hsts-behind-proxy` - `redirect-to-https` - `ssl-redirect` - `http-redirect-code` -------------------------------------------------------------------------------- title: "Path matching" url: https://docs.nginx.com/nginx-ingress-controller/configuration/path-matching/index.md -------------------------------------------------------------------------------- This document describes how F5 NGINX Ingress Controller translates VirtualServer and VirtualServerRoute route paths into NGINX [location](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) directives, with a focus on the longest prefix match (`^~`) modifier. ## Overview Each route in a VirtualServer or subroute in a VirtualServerRoute has a `path` field that maps directly to an NGINX `location` directive. NGINX Ingress Controller supports five path matching types, each with different syntax and priority behavior. Understanding these types, especially the longest prefix match, allows you to control exactly which location block handles a given request. ## Matching types and priority The following table lists all five supported path types in order of priority from highest to lowest: | Priority | Path type | Syntax | NGINX directive | Description | | --- | --- | --- | --- | --- | | 1 (highest) | Exact match | `=/path` | `location = /path` | Matches only the exact URI. Stops all further searching immediately. | | 2 | Longest prefix match | `^~/path` | `location ^~ /path` | Matches by prefix and stops NGINX from evaluating any regex locations. | | 3 | Regex (case-sensitive) | `~ pattern` | `location ~ "pattern"` | Matches the URI against a case-sensitive regular expression. | | 3 | Regex (case-insensitive) | `~* pattern` | `location ~* "pattern"` | Matches the URI against a case-insensitive regular expression. | | 4 (lowest) | Prefix match | `/path` | `location /path` | Matches by prefix, but can be overridden by a regex match. | When NGINX receives a request, it evaluates locations in the following order: 1. **Exact match (`=`)**: If the URI matches an exact location, NGINX uses it immediately and stops searching. 2. **Prefix scan**: NGINX scans all prefix locations (both `^~` and plain `/`) and remembers the longest match. 3. **Longest prefix match (`^~`)**: If the longest matching prefix location has the `^~` modifier, NGINX uses it immediately and skips all regex evaluation. 4. **Regex evaluation (`~` and `~*`)**: NGINX checks regex locations in the order they appear in the configuration. The first matching regex wins. 5. **Prefix fallback**: If no regex matches, NGINX uses the longest matching prefix location from step 2. ## Longest prefix match (`^~`) The `^~` modifier gives a prefix location higher priority than regex locations. When a request URI matches a `^~` location, NGINX uses that location immediately and does not evaluate any regular expression locations, even if a regex would also match. This is useful when you want to serve a specific path prefix with certainty, regardless of any regex patterns that might also match. ### When to use longest prefix match Use `^~` when: - You need to guarantee that a specific path prefix is always handled by a particular upstream or action, even if regex locations exist that could match the same URI. - You are serving static assets from a known path (such as `/images/static/` or `/assets/`) and want to prevent regex patterns (such as `~ \.jpg$`) from intercepting those requests. ### Path syntax rules A longest prefix match path must follow these rules: - The path must start with `^~` followed by a valid URI path starting with `/`. - Optional whitespace between `^~` and the path is allowed (for example, `^~ /images/` and `^~/images/` are equivalent). - The URI portion must not contain whitespace, `{`, `}`, `;`, or `\` characters. - The URI portion must not contain `..` path segments. ### How it differs from a regular prefix match | Behavior | Prefix (`/path`) | Longest prefix (`^~/path`) | | --- | --- | --- | | Matches by URI prefix | Yes | Yes | | Can be overridden by regex | Yes | No | | Supports `proxy.rewritePath` | Yes | Yes | | Supports subroute delegation | Yes | Yes | The only difference is that `^~` prevents regex locations from overriding the match. ## VirtualServer example The following VirtualServer defines routes using several path types, including the longest prefix match: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: path-matching spec: host: path-matching.example.com upstreams: - name: static-backend service: static-svc port: 80 - name: images-backend service: images-svc port: 80 - name: app-backend service: app-svc port: 80 routes: - path: "=/images/logo.jpg" action: pass: static-backend - path: "^~/images/static/" action: pass: static-backend - path: "~ \\.jpg$" action: pass: images-backend - path: /images/ action: pass: app-backend ``` This configuration produces the following NGINX location blocks: ```nginx location = /images/logo.jpg { # Exact match - highest priority ... } location ^~ /images/static/ { # Longest prefix match - blocks regex evaluation ... } location ~ "\.jpg$" { # Case-sensitive regex ... } location /images/ { # Regular prefix - lowest priority ... } ``` With this configuration: | Request URI | Matching location | Reason | | --- | --- | --- | | `/images/logo.jpg` | `= /images/logo.jpg` | Exact match has highest priority. | | `/images/static/photo.jpg` | `^~ /images/static/` | Longest prefix match blocks regex (`~ \.jpg$`) from being evaluated. | | `/images/photo.jpg` | `~ "\.jpg$"` | Regular prefix `/images/` matches, but regex takes priority over plain prefix. | | `/images/photo.gif` | `/images/` | No regex matches `.gif`, so the prefix fallback is used. | ## VirtualServerRoute subroutes with longest prefix match When a VirtualServer route delegates to a VirtualServerRoute using the `route` or `routeSelector` field, the parent route's path acts as a constraint on subroute paths. For a `^~` parent path, each subroute must also use the `^~` modifier. ### VirtualServer with delegation ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: path-matching-vsr spec: host: path-matching-vsr.example.com routes: - path: "^~/static/" route: static-routes ``` ### VirtualServerRoute with `^~` subroutes ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServerRoute metadata: name: static-routes spec: host: path-matching-vsr.example.com subroutes: - path: "^~/static/css/" action: pass: css-backend - path: "^~/static/js/" action: pass: js-backend ``` This produces two NGINX location blocks: ```nginx location ^~ /static/css/ { ... } location ^~ /static/js/ { ... } ``` **important:** The parent route's path (`^~/static/`) does not create an NGINX location block. Only the subroute paths become NGINX locations. If a request matches the parent prefix but not any subroute (for example, `/static/fonts/bold.woff`), no location handles it and NGINX returns a 404 response. ## Subroute path constraints When a VirtualServer route delegates to a VirtualServerRoute, the subroute paths must follow constraints based on the parent route's path type: | Parent path type | Subroute requirement | Multiple subroutes | Example | | --- | --- | --- | --- | | Prefix (`/path`) | Each subroute must start with the parent path and use the prefix type. | Yes | Parent: `/images/`, subroutes: `/images/thumbnails/`, `/images/originals/` | | Longest prefix (`^~/path`) | Each subroute must start with the parent path string including the `^~` modifier. | Yes | Parent: `^~/static/`, subroutes: `^~/static/css/`, `^~/static/js/` | | Regex (`~/pattern` or `~*/pattern`) | The subroute must have the exact same path as the parent. | No (one only) | Parent: `~ \.jpg$`, subroute: `~ \.jpg$` | | Exact (`=/path`) | The subroute must have the exact same path as the parent. | No (one only) | Parent: `=/exact`, subroute: `=/exact` | For longest prefix parent paths, a plain prefix subroute like `/static/css/` is rejected because the validation requires that the subroute path string starts with `^~/static/`. The `^~` modifier must be present on every subroute. ## See also - [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) for the full CRD field reference. - NGINX [location directive documentation](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) for the upstream NGINX matching algorithm. - NGINX [location priority](/nginx/admin-guide/web-server/web-server.md#nginx-location-priority) for a detailed explanation of NGINX's location evaluation order. - [Ingress path matching using path-regex annotation](/nic/tutorials/ingress-path-regex-annotation.md) for configuring regex path matching with Ingress resources. - [Path matching examples](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/path-matching) on GitHub for runnable examples of all five path types. -------------------------------------------------------------------------------- title: "Host and Listener collisions" url: https://docs.nginx.com/nginx-ingress-controller/configuration/host-and-listener-collisions/index.md -------------------------------------------------------------------------------- This document explains how F5 NGINX Ingress Controller handles host and listener collisions between resources. --- ## Winner Selection Algorithm If multiple resources contend for the same host or listener, NGINX Ingress Controller will pick the winner based on the `creationTimestamp` of the resources: the oldest resource will win. In case there are more than one oldest resource (their `creationTimestamp` is the same), NGINX Ingress Controller will choose the resource with the lexicographically smallest `uid`. **Note:** The `creationTimestamp` and `uid` fields are part of the [ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/) resource. --- ## Host collisions A host collision occurs when multiple Ingress, VirtualServer, and TransportServer (configured for TLS Passthrough) resources configure the same `host`. When [`-allow-empty-ingress-host`](/nic/configuration/global-configuration/command-line-arguments.md#cmdoption-allow-empty-ingress-host) is enabled, multiple Ingress resources without a `host` also constitute a host collision for the default server slot. NGINX Ingress Controller has two strategies for handling host collisions: - Choosing a single "winner" resource to handle the host. - Merging the configuration of the conflicting resources. --- ### Choosing the winner Consider the following two resources: - `cafe-ingress` Ingress: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress spec: ingressClassName: nginx rules: - host: cafe.example.com . . . ``` - `cafe-virtual-server` VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe-virtual-server spec: host: cafe.example.com . . . ``` If a user creates both resources in the cluster, a host collision will occur. NGINX Ingress Controller will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). If `cafe-virtual-server` was created first, it will win the host `cafe.example.com` and NGINX Ingress Controller will reject `cafe-ingress`. This will be reflected in the events and in the resource's status field: ```shell kubectl describe vs cafe-virtual-server ``` ```text ... Status: ... Message: Configuration for default/cafe-virtual-server was added or updated Reason: AddedOrUpdated State: Valid Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 9s nginx-ingress-controller Configuration for default/cafe-virtual-server was added or updated ``` ```shell kubectl describe ingress cafe-ingress ``` ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 66s nginx-ingress-controller All hosts are taken by other resources ``` Similarly, if `cafe-ingress` was created first, it will win `cafe.example.com` and NGINX Ingress Controller will reject `cafe-virtual-server`. **Note:** You can configure multiple hosts for Ingress resources, and its possible that an Ingress resource can be the winner for some of its hosts and a loser for the others. For example, if `cafe-ingress` had an additional rule host rule for `pub.example.com`, NGINX Ingress Controller would not reject the Ingress. Instead, it would allow `cafe-ingress` to handle `pub.example.com`. --- ### Merging configuration for the same host It is possible to merge configuration for multiple Ingress resources for the same host. One common use case for this approach is distributing resources across multiple namespaces. The [Cross-namespace configuration](/nic/configuration/ingress-resources/cross-namespace-configuration.md) topic has more information. It is *not* possible to merge the configurations for multiple VirtualServer resources for the same host. However, you can split the VirtualServers into multiple VirtualServerRoute resources, which a single VirtualServer can then reference. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/cross-namespace-configuration) on GitHub. It is *not* possible to merge configuration for multiple TransportServer resources. --- ## Listener/Host collisions Listener/Host collisions occur when multiple TransportServer resources (configured for TCP/UDP load balancing) specify the same combination of `spec.listener.name` and `spec.host`. The combination of `spec.listener.name` and `spec.host` must be unique among all TransportServer resources. If two TransportServer resources specify the same spec.listener.name and spec.host, one of them will be rejected to prevent conflicts. In the case where spec.host is not specified, it is considered an empty string. NGINX Ingress Controller will choose the winner, which will own that listener and host combination. --- ### Choosing the winner Consider the following two resources: - `tcp-1` TransportServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: tcp-1 spec: host: dns.example.com listener: name: dns-tcp protocol: TCP . . . ``` - `tcp-2` TransportServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: tcp-2 spec: host: dns.example.com listener: name: dns-tcp protocol: TCP . . . ``` If a user creates both resources in the cluster, a listener collision will occur. As a result, NGINX Ingress Controller will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). In our example, if `tcp-1` was created first, it will win the listener `dns-tcp` and NGINX Ingress Controller will reject `tcp-2`. This will be reflected in the events and in the resource's status field: ```shell kubectl describe ts tcp-2 ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 10s nginx-ingress-controller Listener dns-tcp is taken by another resource ``` Similarly, if `tcp-2` was created first, it will win `dns-tcp` and NGINX Ingress Controller will reject `tcp-1`. -------------------------------------------------------------------------------- title: "Host and Listener collisions" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/host-and-listener-collisions/index.md -------------------------------------------------------------------------------- This document explains how F5 NGINX Ingress Controller LTS handles host and listener collisions between resources. --- ## Winner Selection Algorithm If multiple resources contend for the same host or listener, NGINX Ingress Controller LTS will pick the winner based on the `creationTimestamp` of the resources: the oldest resource will win. In case there are more than one oldest resource (their `creationTimestamp` is the same), NGINX Ingress Controller LTS will choose the resource with the lexicographically smallest `uid`. **note:** The `creationTimestamp` and `uid` fields are part of the [ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/) resource. --- ## Host collisions A host collision occurs when multiple Ingress, VirtualServer, and TransportServer (configured for TLS Passthrough) resources configure the same `host`. NGINX Ingress Controller LTS has two strategies for handling host collisions: - Choosing a single "winner" resource to handle the host. - Merging the configuration of the conflicting resources. --- ### Choosing the winner Consider the following two resources: - `cafe-ingress` Ingress: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress spec: ingressClassName: nginx rules: - host: cafe.example.com . . . ``` - `cafe-virtual-server` VirtualServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: cafe-virtual-server spec: host: cafe.example.com . . . ``` If a user creates both resources in the cluster, a host collision will occur. NGINX Ingress Controller LTS will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). If `cafe-virtual-server` was created first, it will win the host `cafe.example.com` and NGINX Ingress Controller LTS will reject `cafe-ingress`. This will be reflected in the events and in the resource's status field: ```shell kubectl describe vs cafe-virtual-server ``` ```text ... Status: ... Message: Configuration for default/cafe-virtual-server was added or updated Reason: AddedOrUpdated State: Valid Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedOrUpdated 9s nginx-ingress-controller Configuration for default/cafe-virtual-server was added or updated ``` ```shell kubectl describe ingress cafe-ingress ``` ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 66s nginx-ingress-controller All hosts are taken by other resources ``` Similarly, if `cafe-ingress` was created first, it will win `cafe.example.com` and NGINX Ingress Controller LTS will reject `cafe-virtual-server`. **note:** You can configure multiple hosts for Ingress resources, and its possible that an Ingress resource can be the winner for some of its hosts and a loser for the others. For example, if `cafe-ingress` had an additional rule host rule for `pub.example.com`, NGINX Ingress Controller LTS would not reject the Ingress. Instead, it would allow `cafe-ingress` to handle `pub.example.com`. --- ### Merging configuration for the same host It is possible to merge configuration for multiple Ingress resources for the same host. One common use case for this approach is distributing resources across multiple namespaces. The [Cross-namespace configuration](/nic/lts/configuration/ingress-resources/cross-namespace-configuration.md) topic has more information. It is *not* possible to merge the configurations for multiple VirtualServer resources for the same host. However, you can split the VirtualServers into multiple VirtualServerRoute resources, which a single VirtualServer can then reference. See the [corresponding example](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/cross-namespace-configuration) on GitHub. It is *not* possible to merge configuration for multiple TransportServer resources. --- ## Listener/Host collisions Listener/Host collisions occur when multiple TransportServer resources (configured for TCP/UDP load balancing) specify the same combination of `spec.listener.name` and `spec.host`. The combination of `spec.listener.name` and `spec.host` must be unique among all TransportServer resources. If two TransportServer resources specify the same spec.listener.name and spec.host, one of them will be rejected to prevent conflicts. In the case where spec.host is not specified, it is considered an empty string. NGINX Ingress Controller LTS will choose the winner, which will own that listener and host combination. --- ### Choosing the winner Consider the following two resources: - `tcp-1` TransportServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: tcp-1 spec: host: dns.example.com listener: name: dns-tcp protocol: TCP . . . ``` - `tcp-2` TransportServer: ```yaml apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: tcp-2 spec: host: dns.example.com listener: name: dns-tcp protocol: TCP . . . ``` If a user creates both resources in the cluster, a listener collision will occur. As a result, NGINX Ingress Controller LTS will pick the winner using the [winner selection algorithm](#winner-selection-algorithm). In our example, if `tcp-1` was created first, it will win the listener `dns-tcp` and NGINX Ingress Controller LTS will reject `tcp-2`. This will be reflected in the events and in the resource's status field: ```shell kubectl describe ts tcp-2 ``` ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Rejected 10s nginx-ingress-controller Listener dns-tcp is taken by another resource ``` Similarly, if `tcp-2` was created first, it will win `dns-tcp` and NGINX Ingress Controller LTS will reject `tcp-1`. -------------------------------------------------------------------------------- title: "Run multiple NGINX Ingress Controller LTSs" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/multiple-controllers/index.md -------------------------------------------------------------------------------- This document describes how to run multiple F5 NGINX Ingress Controller LTS instances. It explains the following topics: - Ingress class concept. - How to run NGINX Ingress Controller LTS in the same cluster with another Ingress Controller and prevent conflicts between them - How to run multiple NGINX Ingress Controller LTSs. **note:** This document refers to [Ingress](/nic/lts/configuration/ingress-resources/basic-configuration.md), [VirtualServer](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserver-specification), [VirtualServerRoute](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroute-specification), and [TransportServer](/nic/lts/configuration/transportserver-resource.md) resources as "configuration resources". ## Ingress class The [IngressClass](https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class) resource allows for multiple Ingress Controller to operate in the same cluster. It also allow developers to select which Ingress Controller implementation to use for their Ingress resource. The IngressClass has the following characteristics: - Every Ingress Controller must only handle Ingress resources for its particular class. - Ingress resources need to have the `ingressClassName` field set to the value of the class of the Ingress Controller the user wants to use. - VirtualServer, VirtualServerRoute, Policy, and TransportServer resources need to have the `ingressClassName` field set to the value of the class of the Ingress Controller the user wants to use. ### Configuring Ingress class The default Ingress class of NGINX Ingress Controller LTS is `nginx`, which means that it only handles configuration resources with the Ingress class set to `nginx`. You can customize the class through the `-ingress-class` command-line argument. **note:** If the class of an Ingress resource is not set, Kubernetes will set it to the class of the default Ingress Controller. To make the Ingress Controller the default one, the `ingressclass.kubernetes.io/is-default-class` property must be set on the IngressClass resource. To learn more, see Step 3 *Create an IngressClass resource* of the [Create Common Resources](/nic/lts/install/manifests.md#create-common-resources) section. For VirtualServer, VirtualServerRoute, Policy and TransportServer resources, NGINX Ingress Controller LTS will always handle resources with an empty class. ## Run NGINX Ingress Controller LTS and another Ingress Controller It is possible to run NGINX Ingress Controller LTS and an Ingress Controller for another load balancer in the same cluster. This is often the case if you create your cluster through a cloud provider's managed Kubernetes service that by default might include the Ingress Controller for the HTTP load balancer of the cloud provider, and you want to use NGINX Ingress Controller LTS. To make sure that NGINX Ingress Controller LTS handles specific configuration resources, update those resources with the class set to the value that is configured in NGINX Ingress Controller LTS. By default, this is `nginx`. ## Run multiple NGINX Ingress Controller LTSs When running NGINX Ingress Controller LTS, you have the following options with regards to which configuration resources it handles: - Cluster-wide Ingress Controller (default): NGINX Ingress Controller LTS handles configuration resources created in any namespace of the cluster. As NGINX is a high-performance load balancer capable of serving many applications at the same time, this option is used by default in our installation manifests and Helm chart. - Defined-namespace Ingress Controller: You can configure the Ingress Controller to handle configuration resources only from particular namespaces, which is controlled through the `-watch-namespace` command-line argument. This can be useful if you want to use different NGINX Ingress Controller LTSs for different applications, both in terms of isolation and/or operation. - Ingress Controller for Specific Ingress Class: This option works in conjunction with either of the options above. You can further customize which configuration resources are handled by the Ingress Controller by configuring the class of the Ingress Controller and using that class in your configuration resources. The [Configuring Ingress Class](#configuring-ingress-class) section above explains where. These options allow you to run multiple NGINX Ingress Controller LTSs, each handling a different set of configuration resources. **note:** There is a reference available for all [Command-line arguments](/nic/lts/configuration/global-configuration/command-line-arguments) Every command-line argument is also available as a parameter for the [Helm chart](/nic/lts/install/helm/parameters.md) -------------------------------------------------------------------------------- title: "Run multiple NGINX Ingress Controllers" url: https://docs.nginx.com/nginx-ingress-controller/install/multiple-controllers/index.md -------------------------------------------------------------------------------- This document describes how to run multiple F5 NGINX Ingress Controller instances. It explains the following topics: - Ingress class concept. - How to run NGINX Ingress Controller in the same cluster with another Ingress Controller and prevent conflicts between them - How to run multiple NGINX Ingress Controllers. **Note:** This document refers to [Ingress](/nic/configuration/ingress-resources/basic-configuration.md), [VirtualServer](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserver-specification), [VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources.md#virtualserverroute-specification), and [TransportServer](/nic/configuration/transportserver-resource.md) resources as "configuration resources". ## Ingress class The [IngressClass](https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class) resource allows for multiple Ingress Controller to operate in the same cluster. It also allow developers to select which Ingress Controller implementation to use for their Ingress resource. The IngressClass has the following characteristics: - Every Ingress Controller must only handle Ingress resources for its particular class. - Ingress resources need to have the `ingressClassName` field set to the value of the class of the Ingress Controller the user wants to use. - VirtualServer, VirtualServerRoute, Policy, and TransportServer resources need to have the `ingressClassName` field set to the value of the class of the Ingress Controller the user wants to use. ### Configuring Ingress class The default Ingress class of NGINX Ingress Controller is `nginx`, which means that it only handles configuration resources with the Ingress class set to `nginx`. You can customize the class through the `-ingress-class` command-line argument. **Note:** If the class of an Ingress resource is not set, Kubernetes will set it to the class of the default Ingress Controller. To make the Ingress Controller the default one, the `ingressclass.kubernetes.io/is-default-class` property must be set on the IngressClass resource. To learn more, see Step 3 *Create an IngressClass resource* of the [Create Common Resources](/nic/install/manifests.md#create-common-resources) section. For VirtualServer, VirtualServerRoute, Policy and TransportServer resources, NGINX Ingress Controller will always handle resources with an empty class. ## Run NGINX Ingress Controller and another Ingress Controller It is possible to run NGINX Ingress Controller and an Ingress Controller for another load balancer in the same cluster. This is often the case if you create your cluster through a cloud provider's managed Kubernetes service that by default might include the Ingress Controller for the HTTP load balancer of the cloud provider, and you want to use NGINX Ingress Controller. To make sure that NGINX Ingress Controller handles specific configuration resources, update those resources with the class set to the value that is configured in NGINX Ingress Controller. By default, this is `nginx`. ## Run multiple NGINX Ingress Controllers When running NGINX Ingress Controller, you have the following options with regards to which configuration resources it handles: - Cluster-wide Ingress Controller (default): NGINX Ingress Controller handles configuration resources created in any namespace of the cluster. As NGINX is a high-performance load balancer capable of serving many applications at the same time, this option is used by default in our installation manifests and Helm chart. - Defined-namespace Ingress Controller: You can configure the Ingress Controller to handle configuration resources only from particular namespaces, which is controlled through the `-watch-namespace` command-line argument. This can be useful if you want to use different NGINX Ingress Controllers for different applications, both in terms of isolation and/or operation. - Ingress Controller for Specific Ingress Class: This option works in conjunction with either of the options above. You can further customize which configuration resources are handled by the Ingress Controller by configuring the class of the Ingress Controller and using that class in your configuration resources. The [Configuring Ingress Class](#configuring-ingress-class) section above explains where. These options allow you to run multiple NGINX Ingress Controllers, each handling a different set of configuration resources. **Note:** There is a reference available for all [Command-line arguments](/nic/configuration/global-configuration/command-line-arguments) Every command-line argument is also available as a parameter for the [Helm chart](/nic/install/helm/parameters.md) -------------------------------------------------------------------------------- title: "Proxy Buffer Configuration Auto-Adjustment" url: https://docs.nginx.com/nginx-ingress-controller/configuration/proxy-buffers-configuration/index.md -------------------------------------------------------------------------------- This document explains how the `--enable-directive-autoadjust` option prevents NGINX configuration errors by automatically adjusting some HTTP proxy buffer directives. --- ## What it does The `--enable-directive-autoadjust` feature automatically fixes common proxy buffer configuration mistakes that would otherwise cause NGINX to fail with errors like: ```text [emerg] "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer ``` **What gets fixed:** - If you don't specify `proxy_buffers`, it sets a sensible default of `8 4k` - If your `proxy_busy_buffers_size` is too large, it reduces it to a safe value - If the number of proxy buffers is outside the valid range (minimum 2, maximum 1024), it gets clamped to those limits - Empty or invalid buffer settings get corrected automatically **Works with:** - [ConfigMap settings](/nic/configuration/global-configuration/configmap-resource.md#general-customization) - [Ingress annotations](/nic/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) - [VirtualServer upstream buffer configurations](/nic/configuration/virtualserver-and-virtualserverroute-resources/#upstream) --- ## How to enable auto-adjustment #### Manifests Add the flag to the controller container: ```yaml args: - --enable-directive-autoadjust=true ``` #### Helm Enable via the Helm chart values file: ```yaml controller: directiveAutoAdjust: "true" ``` --- ## Examples ### Example 1 **Input:** ```yaml data: proxy-buffer-size: "5m" proxy-buffers: "8 1m" ``` #### Before (Error) Before enabling `--enable-directive-autoadjust`, NGINX fails to start with configuration validation errors. ```shell stderr: "2025/08/26 14:29:49 [emerg] 196#196: "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer in /etc/nginx/nginx.conf:121" ``` #### After (Fixed) With `--enable-directive-autoadjust`, the configuration is automatically adjusted: ```nginx proxy_buffers 8 1m; proxy_buffer_size 5m; proxy_busy_buffers_size 5m; ``` Logs: ```text I20250826 14:31:54.515490 1 configmaps.go:380] Changes made to proxy values: adjusted proxy_busy_buffers_size from to 5m because it was too small ``` ### Example 2 **Input:** ```yaml data: proxy-buffers: "1000000 1m" # Extremely high buffer count proxy-buffer-size: "999m" # Very large buffer size proxy-busy-buffers-size: "500m" ``` #### Before (Error) ```shell stderr: "2025/08/26 14:34:46 [emerg] 47#47: "proxy_busy_buffers_size" must be equal to or greater than the maximum of the value of "proxy_buffer_size" and one of the "proxy_buffers" in /etc/nginx/nginx.conf:121\n" ``` #### After (Fixed) With `--enable-directive-autoadjust`, sensible defaults are applied: ```shell proxy_buffers 1024 1m; proxy_buffer_size 999m; proxy_busy_buffers_size 999m; ``` Logs: ```shell I20250826 14:36:47.864375 1 configmaps.go:380] Changes made to proxy values: adjusted proxy_buffers number from 1000000 to 1024 I20250826 14:36:47.864389 1 configmaps.go:380] Changes made to proxy values: adjusted proxy_busy_buffers_size from 500m to 999m because it was too small ``` --- ## Monitoring and logging The controller outputs a log message whenever any of the proxy buffer directives are changed. Examples: ```text I20250826 14:06:43.734757 1 annotations.go:341] Changes made to proxy values: adjusted proxy_buffer_size from 512k to 64k because it was too big for proxy_buffers (2 64k) I20250826 14:06:43.734842 1 annotations.go:341] Changes made to proxy values: adjusted proxy_busy_buffers_size from to 64k because it was too small ``` View adjustment logs: ```bash kubectl logs -n | grep "Changes made to proxy values" ``` -------------------------------------------------------------------------------- title: "Proxy Buffer Configuration Auto-Adjustment" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/proxy-buffers-configuration/index.md -------------------------------------------------------------------------------- This document explains how the `--enable-directive-autoadjust` option prevents NGINX configuration errors by automatically adjusting some HTTP proxy buffer directives. --- ## What it does The `--enable-directive-autoadjust` feature automatically fixes common proxy buffer configuration mistakes that would otherwise cause NGINX to fail with errors like: ```text [emerg] "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer ``` **What gets fixed:** - If you don't specify `proxy_buffers`, it sets a sensible default of `8 4k` - If your `proxy_busy_buffers_size` is too large, it reduces it to a safe value - If the number of proxy buffers is outside the valid range (minimum 2, maximum 1024), it gets clamped to those limits - Empty or invalid buffer settings get corrected automatically **Works with:** - [ConfigMap settings](/nic/lts/configuration/global-configuration/configmap-resource.md#general-customization) - [Ingress annotations](/nic/lts/configuration/ingress-resources/advanced-configuration-with-annotations/#general-customization) - [VirtualServer upstream buffer configurations](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources/#upstream) --- ## How to enable auto-adjustment #### Manifests Add the flag to the controller container: ```yaml args: - --enable-directive-autoadjust=true ``` #### Helm Enable via the Helm chart values file: ```yaml controller: directiveAutoAdjust: "true" ``` --- ## Examples ### Example 1 **Input:** ```yaml data: proxy-buffer-size: "5m" proxy-buffers: "8 1m" ``` #### Before (Error) Before enabling `--enable-directive-autoadjust`, NGINX fails to start with configuration validation errors. ```shell stderr: "2025/08/26 14:29:49 [emerg] 196#196: "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer in /etc/nginx/nginx.conf:121" ``` #### After (Fixed) With `--enable-directive-autoadjust`, the configuration is automatically adjusted: ```nginx proxy_buffers 8 1m; proxy_buffer_size 5m; proxy_busy_buffers_size 5m; ``` Logs: ```text I20250826 14:31:54.515490 1 configmaps.go:380] Changes made to proxy values: adjusted proxy_busy_buffers_size from to 5m because it was too small ``` ### Example 2 **Input:** ```yaml data: proxy-buffers: "1000000 1m" # Extremely high buffer count proxy-buffer-size: "999m" # Very large buffer size proxy-busy-buffers-size: "500m" ``` #### Before (Error) ```shell stderr: "2025/08/26 14:34:46 [emerg] 47#47: "proxy_busy_buffers_size" must be equal to or greater than the maximum of the value of "proxy_buffer_size" and one of the "proxy_buffers" in /etc/nginx/nginx.conf:121\n" ``` #### After (Fixed) With `--enable-directive-autoadjust`, sensible defaults are applied: ```shell proxy_buffers 1024 1m; proxy_buffer_size 999m; proxy_busy_buffers_size 999m; ``` Logs: ```shell I20250826 14:36:47.864375 1 configmaps.go:380] Changes made to proxy values: adjusted proxy_buffers number from 1000000 to 1024 I20250826 14:36:47.864389 1 configmaps.go:380] Changes made to proxy values: adjusted proxy_busy_buffers_size from 500m to 999m because it was too small ``` --- ## Monitoring and logging The controller outputs a log message whenever any of the proxy buffer directives are changed. Examples: ```text I20250826 14:06:43.734757 1 annotations.go:341] Changes made to proxy values: adjusted proxy_buffer_size from 512k to 64k because it was too big for proxy_buffers (2 64k) I20250826 14:06:43.734842 1 annotations.go:341] Changes made to proxy values: adjusted proxy_busy_buffers_size from to 64k because it was too small ``` View adjustment logs: ```bash kubectl logs -n | grep "Changes made to proxy values" ``` -------------------------------------------------------------------------------- title: "Deploy a Policy for access control" url: https://docs.nginx.com/nginx-ingress-controller/configuration/access-control/index.md -------------------------------------------------------------------------------- This topic describes how to use F5 NGINX Ingress Controller to apply and update a Policy for access control. You can use access control policies with [VirtualServer custom resources](/nic/configuration/virtualserver-and-virtualserverroute-resources.md) or with [Ingress resources](/nic/configuration/ingress-resources/basic-configuration.md) using the `nginx.org/policies` annotation. --- ## Before you begin You should have a [working NGINX Ingress Controller](/nic/install/helm.md) instance. For ease of use in shell commands, set the following shell variables: 1. The public IP address for your NGINX Ingress Controller instance. ```shell IC_IP= ``` 2. The HTTP port of the same instance. ```shell IC_HTTP_PORT= ``` 3. The HTTPS port of the same instance (used for the [Ingress resource example](#use-access-control-with-ingress-resources)). ```shell IC_HTTPS_PORT= ``` --- ## Use access control with VirtualServer resources ### Deploy the example application Create the file _webapp.yaml_ with the following contents: *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/webapp.yaml]* Apply it using `kubectl`: ```shell kubectl apply -f webapp.yaml ``` --- ### Deploy a Policy to create a deny rule Create a file named _access-control-policy-deny.yaml_. The highlighted _deny_ field will be used by the example application, and should be changed to the subnet of your machine. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/access-control-policy-deny.yaml]* Apply the policy: ```shell kubectl apply -f access-control-policy-deny.yaml ``` --- ### Configure load balancing Create a file named _virtual-server.yaml_ for the VirtualServer resource. The _policies_ field references the access control Policy created in the previous section. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/virtual-server.yaml]* Apply the policy: ```shell kubectl apply -f virtual-server.yaml ``` --- ## Test the example application Use `curl` to attempt to access the application: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT ``` ```text 403 Forbidden

403 Forbidden

``` The *403* response is expected, successfully blocking your machine. --- ### Update the Policy to create an allow rule Update the Policy with the file _access-control-policy-allow.yaml_, setting the _allow_ field to the subnet of your machine. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/custom-resources/access-control/access-control-policy-allow.yaml]* Apply the Policy: ```shell kubectl apply -f access-control-policy-allow.yaml ``` ---- ### Verify the Policy update Attempt to access the application again: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT ``` ```text Server address: 10.64.0.13:8080 Server name: webapp-5cbbc7bd78-wf85w ``` The successful response demonstrates that the policy has been updated. --- ## Use access control with Ingress resources You can also apply access control policies to standard Kubernetes Ingress resources using the `nginx.org/policies` annotation. This section walks through a complete example. ### Deploy the cafe application Create the file _cafe.yaml_ with the following contents: *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/access-control/cafe.yaml]* Apply it using `kubectl`: ```shell kubectl apply -f cafe.yaml ``` ### Configure NGINX to use the X-Real-IP header Create the file _nginx-config.yaml_ to configure NGINX to trust the `X-Real-IP` header. This ensures the access control policy uses the client IP provided in that header. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/access-control/nginx-config.yaml]* Apply the ConfigMap: ```shell kubectl apply -f nginx-config.yaml ``` ### Deploy a Policy to create an allow rule Create a file named _access-control-policy-allow.yaml_. The highlighted _allow_ field permits traffic from the `10.0.0.0/8` CIDR range and blocks all other addresses. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/access-control/access-control-policy-allow.yaml]* Apply the policy: ```shell kubectl apply -f access-control-policy-allow.yaml ``` ### Create the Ingress resource Create a file named _cafe-ingress.yaml_ for the Ingress resource. The highlighted `nginx.org/policies` annotation references the access control Policy created in the previous step. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/access-control/cafe-ingress.yaml]* Apply the Ingress: ```shell kubectl apply -f cafe-ingress.yaml ``` ### Test the allow policy 1. Send a request with an IP in the allowed `10.0.0.0/8` range using the `X-Real-IP` header: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 10.0.0.1" ``` ```text Server address: 10.244.0.6:8080 Server name: coffee-7586895968-r26zn ... ``` The request succeeds because `10.0.0.1` is in the allowed range. 2. Send a request with an IP outside the allowed range: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 192.168.1.1" ``` ```text 403 Forbidden

403 Forbidden

``` The *403* response confirms that NGINX blocks clients outside the allowed range. ### Update the Policy to create a deny rule Update the Policy with the file _access-control-policy-deny.yaml_, which denies traffic from the `10.0.0.0/8` CIDR range and allows all other addresses. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/main/examples/ingress-resources/access-control/access-control-policy-deny.yaml]* Apply the updated Policy: ```shell kubectl apply -f access-control-policy-deny.yaml ``` The Ingress resource picks up the change automatically because the policy name (`webapp-policy`) stays the same. ### Verify the deny policy 1. Send a request with an IP in the now-denied `10.0.0.0/8` range: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 10.0.0.1" ``` ```text 403 Forbidden

403 Forbidden

``` The same IP that was previously allowed is now rejected. 2. Send a request with an IP outside the denied range: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 192.168.1.1" ``` ```text Server address: 10.244.0.6:8080 Server name: coffee-7586895968-r26zn ... ``` Clients outside the denied range are now allowed through. -------------------------------------------------------------------------------- title: "Deploy a Policy for access control" url: https://docs.nginx.com/nginx-ingress-controller/lts/configuration/access-control/index.md -------------------------------------------------------------------------------- This topic describes how to use F5 NGINX Ingress Controller LTS to apply and update a Policy for access control. You can use access control policies with [VirtualServer custom resources](/nic/lts/configuration/virtualserver-and-virtualserverroute-resources.md) or with [Ingress resources](/nic/lts/configuration/ingress-resources/basic-configuration.md) using the `nginx.org/policies` annotation. --- ## Before you begin You should have a [working NGINX Ingress Controller LTS](/nic/lts/install/helm.md) instance. For ease of use in shell commands, set the following shell variables: 1. The public IP address for your NGINX Ingress Controller LTS instance. ```shell IC_IP= ``` 2. The HTTP port of the same instance. ```shell IC_HTTP_PORT= ``` 3. The HTTPS port of the same instance (used for the [Ingress resource example](#use-access-control-with-ingress-resources)). ```shell IC_HTTPS_PORT= ``` --- ## Use access control with VirtualServer resources ### Deploy the example application Create the file _webapp.yaml_ with the following contents: *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/custom-resources/access-control/webapp.yaml]* Apply it using `kubectl`: ```shell kubectl apply -f webapp.yaml ``` --- ### Deploy a Policy to create a deny rule Create a file named _access-control-policy-deny.yaml_. The highlighted _deny_ field will be used by the example application, and should be changed to the subnet of your machine. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/custom-resources/access-control/access-control-policy-deny.yaml]* Apply the policy: ```shell kubectl apply -f access-control-policy-deny.yaml ``` --- ### Configure load balancing Create a file named _virtual-server.yaml_ for the VirtualServer resource. The _policies_ field references the access control Policy created in the previous section. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/custom-resources/access-control/virtual-server.yaml]* Apply the policy: ```shell kubectl apply -f virtual-server.yaml ``` --- ## Test the example application Use `curl` to attempt to access the application: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT ``` ```text 403 Forbidden

403 Forbidden

``` The *403* response is expected, successfully blocking your machine. --- ### Update the Policy to create an allow rule Update the Policy with the file _access-control-policy-allow.yaml_, setting the _allow_ field to the subnet of your machine. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/custom-resources/access-control/access-control-policy-allow.yaml]* Apply the Policy: ```shell kubectl apply -f access-control-policy-allow.yaml ``` ---- ### Verify the Policy update Attempt to access the application again: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP http://webapp.example.com:$IC_HTTP_PORT ``` ```text Server address: 10.64.0.13:8080 Server name: webapp-5cbbc7bd78-wf85w ``` The successful response demonstrates that the policy has been updated. --- ## Use access control with Ingress resources You can also apply access control policies to standard Kubernetes Ingress resources using the `nginx.org/policies` annotation. This section walks through a complete example. ### Deploy the cafe application Create the file _cafe.yaml_ with the following contents: *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/ingress-resources/access-control/cafe.yaml]* Apply it using `kubectl`: ```shell kubectl apply -f cafe.yaml ``` ### Configure NGINX to use the X-Real-IP header Create the file _nginx-config.yaml_ to configure NGINX to trust the `X-Real-IP` header. This ensures the access control policy uses the client IP provided in that header. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/ingress-resources/access-control/nginx-config.yaml]* Apply the ConfigMap: ```shell kubectl apply -f nginx-config.yaml ``` ### Deploy a Policy to create an allow rule Create a file named _access-control-policy-allow.yaml_. The highlighted _allow_ field permits traffic from the `10.0.0.0/8` CIDR range and blocks all other addresses. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/ingress-resources/access-control/access-control-policy-allow.yaml]* Apply the policy: ```shell kubectl apply -f access-control-policy-allow.yaml ``` ### Create the Ingress resource Create a file named _cafe-ingress.yaml_ for the Ingress resource. The highlighted `nginx.org/policies` annotation references the access control Policy created in the previous step. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/ingress-resources/access-control/cafe-ingress.yaml]* Apply the Ingress: ```shell kubectl apply -f cafe-ingress.yaml ``` ### Test the allow policy 1. Send a request with an IP in the allowed `10.0.0.0/8` range using the `X-Real-IP` header: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 10.0.0.1" ``` ```text Server address: 10.244.0.6:8080 Server name: coffee-7586895968-r26zn ... ``` The request succeeds because `10.0.0.1` is in the allowed range. 2. Send a request with an IP outside the allowed range: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 192.168.1.1" ``` ```text 403 Forbidden

403 Forbidden

``` The *403* response confirms that NGINX blocks clients outside the allowed range. ### Update the Policy to create a deny rule Update the Policy with the file _access-control-policy-deny.yaml_, which denies traffic from the `10.0.0.0/8` CIDR range and allows all other addresses. *[Code: https://raw.githubusercontent.com/nginx/kubernetes-ingress/refs/heads/release-2026-lts/examples/ingress-resources/access-control/access-control-policy-deny.yaml]* Apply the updated Policy: ```shell kubectl apply -f access-control-policy-deny.yaml ``` The Ingress resource picks up the change automatically because the policy name (`webapp-policy`) stays the same. ### Verify the deny policy 1. Send a request with an IP in the now-denied `10.0.0.0/8` range: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 10.0.0.1" ``` ```text 403 Forbidden

403 Forbidden

``` The same IP that was previously allowed is now rejected. 2. Send a request with an IP outside the denied range: ```shell curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee --insecure -H "X-Real-IP: 192.168.1.1" ``` ```text Server address: 10.244.0.6:8080 Server name: coffee-7586895968-r26zn ... ``` Clients outside the denied range are now allowed through. -------------------------------------------------------------------------------- title: "Upgrade NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/install/upgrade/index.md -------------------------------------------------------------------------------- This document describes how to upgrade F5 NGINX Ingress Controller when a new version releases. It covers the necessary steps for minor versions as well as major versions (Such as 3.x to 4.x). Many of the nuances in upgrade paths relate to how custom resource definitions (CRDs) are managed. ## Minor NGINX Ingress Controller upgrades ### Upgrade NGINX Ingress Controller CRDs **Note:** If you are running NGINX Ingress Controller v3.x, you should read [Upgrade from NGINX Ingress Controller v3.x to v4.0.0](/nic/install/upgrade.md#upgrade-from-3x-to-4x) before continuing. To upgrade the CRDs, pull the Helm chart source, then use _kubectl apply_: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress --untar --version kubectl apply -f crds/ ``` Alternatively, CRDs can be upgraded without pulling the chart by running: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` In the above command, `v` represents the version of the NGINX Ingress Controller release rather than the Helm chart version. **Note:** The following warning is expected and can be ignored: `Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply`. Check the [release notes](https://www.github.com/nginx/kubernetes-ingress/releases) for a new release for any special upgrade procedures. ### Upgrade NGINX Ingress Controller charts Once the CRDs have been upgraded, you can then upgrade the release chart. The command depends on if you installed the chart using the registry or from source. To upgrade a release named _my-release_, use the following command: #### OCI registry ```shell helm upgrade my-release oci://ghcr.io/nginx/charts/nginx-ingress --version ``` #### Source ```shell helm upgrade my-release . ``` ## Upgrade from 3.x to 4.x **Note:** The instructions in this section are intended only for users upgrading from NGINX Ingress Controller 3.x to 4.0.0. Internal changes meant that backwards compability was not possible, requiring extra steps to upgrade. This section provides step-by-step instructions for upgrading NGINX Ingress Controller from version v3.x to v4.0.0. There are two necessary steps required - Update the `apiVersion` value of custom resources - Configure structured logging. If you want to use NGINX Plus, you will also need to follow the [Create a license Secret](/nic/install/license-secret.md) topic. ### Update custom resource apiVersion If you're using Helm chart version `v2.x`, update your `GlobalConfiguration`, `Policy`, and `TransportServer` resources from `apiVersion: k8s.nginx.org/v1alpha1` to `apiVersion: k8s.nginx.org/v1` before upgrading to NGINX Ingress Controller 4.0.0. If the Helm chart you have been using is `v1.0.2` or earlier (NGINX Ingress Controller `v3.3.2`), upgrade to Helm chart `v1.4.2` (NGINX Ingress Controller `v3.7.2`) before updating your GlobalConfiguration, Policy, and TransportServer resources. The example below shows the change for a Policy resource: you must do the same for all GlobalConfiguration and TransportServer resources. #### Before ```yaml apiVersion: k8s.nginx.org/v1alpha1 kind: Policy metadata: name: rate-limit-policy spec: rateLimit: rate: 1r/s key: ${binary_remote_addr} zoneSize: 10M ``` #### After ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: rate-limit-policy spec: rateLimit: rate: 1r/s key: ${binary_remote_addr} zoneSize: 10M ``` **Note:** If a *GlobalConfiguration*, *Policy* or *TransportServer* resource is deployed with `apiVersion: k8s.nginx.org/v1alpha1`, it will be **deleted** during the upgrade process. After you move the custom resources to `v1`, run the following `kubectl` commands before upgrading to v4.0.0 Custom Resource Definitions (CRDs) to avoid webhook errors caused by leftover `v1alpha1` resources. For details, see [GitHub issue #7010](https://github.com/nginx/kubernetes-ingress/issues/7010). ```shell kubectl patch customresourcedefinitions transportservers.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}' ``` ```shell kubectl patch customresourcedefinitions globalconfigurations.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}' ``` ### Configure structured logging To configure structured logging, you must update your log deployment arguments from an integer to a string. You can also choose different formats for the log output. **Note:** These options apply to NGINX Ingress Controller logs, and do not affect NGINX logs. | **Level arguments** | **Format arguments** | |---------------------|----------------------| | `trace` | `json` | | `debug` | `text` | | `info` | `glog` | | `warning` | | | `error` | | | `fatal` | | #### Helm The Helm value `controller.logLevel` is now a string instead of an integer. To change the rendering of the log format, use the `controller.logFormat` key. ```yaml controller: logLevel: info logFormat: json ``` #### Manifests The command line argument `-v` has been replaced with `-log-level`, and takes a string instead of an integer. The argument `-logtostderr` has also been deprecated. To change the rendering of the log format, use the `-log-format` argument. ```yaml args: - -log-level=info - -log-format=json ``` ### Create License secret If you're using [NGINX Plus](/nic/overview/nginx-plus.md) with NGINX Ingress Controller, you should read the [Create a license Secret](/nic/install/license-secret.md) topic to set up your NGINX Plus license. The topic also contains guidance for [sending reports to NGINX Instance Manager](/nic/install/license-secret.md#nim), which is necessary for air-gapped environments. Earlier versions required usage reporting through the cluster connector. This is no longer needed because it's now built into NGINX Plus. ## Upgrade a version older than v3.1.0 Starting in version 3.1.0, NGINX Ingress Controller uses updated Helm resource names, labels, and annotations to follow Helm best practices. [See the changes.](https://github.com/nginx/kubernetes-ingress/pull/3606) When you upgrade with Helm from a version earlier than 3.1.0, some resources such as `Deployment`, `DaemonSet`, and `Service` are recreated. This causes downtime. To reduce downtime, update all resources to use the new naming convention. The following steps help you do that. **Note:** The following steps apply to both 2.x and 3.0.x releases. The steps you should follow depend on your Helm release name: #### nginx-ingress Use `kubectl describe` on deployment/daemonset to get the `Selector` value: ```shell kubectl describe deployments -n ``` Copy the key=value under `Selector`, such as: ```shell Selector: app=nginx-ingress-nginx-ingress ``` Check out the latest available tag using `git checkout v` Go to `/kubernetes-ingress/charts/nginx-ingress` Update the `selectorLabels: {}` field in the `values.yaml` file located at `/kubernetes-ingress/charts/nginx-ingress` with the copied `Selector` value. ```shell selectorLabels: {app: nginx-ingress-nginx-ingress} ``` Run `helm upgrade` with following arguments set: ```shell --set serviceNameOverride="nginx-ingress-nginx-ingress" --set controller.name="" --set fullnameOverride="nginx-ingress-nginx-ingress" ``` It might look like this: ```shell helm upgrade nginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress --version 0.19.0 --set controller.kind=deployment/daemonset --set controller.nginxplus=false/true --set controller.image.pullPolicy=Always --set serviceNameOverride="nginx-ingress-nginx-ingress" --set controller.name="" --set fullnameOverride="nginx-ingress-nginx-ingress" -f values.yaml ``` Once the upgrade process has finished, use `kubectl describe` on the deployment to verify the change by reviewing its events: ```text Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 9m11s deployment-controller Scaled up replica set nginx-ingress-nginx-ingress- to 1 Normal ScalingReplicaSet 101s deployment-controller Scaled up replica set nginx-ingress-nginx-ingress- to 1 Normal ScalingReplicaSet 98s deployment-controller Scaled down replica set nginx-ingress-nginx-ingress- to 0 from 1 ``` #### Other release names Use `kubectl describe` on deployment/daemonset to get the `Selector` value: ```shell kubectl describe deployment/daemonset -n ``` Copy the key=value under ```Selector```, such as: ```shell Selector: app=-nginx-ingress ``` Check out the latest available tag using `git checkout v` Go to `/kubernetes-ingress/charts/nginx-ingress`. Update the `selectorLabels: {}` field in the `values.yaml` file located at `/kubernetes-ingress/charts/nginx-ingress` with the copied `Selector` value. ```shell selectorLabels: {app: -nginx-ingress} ``` Run `helm upgrade` with following arguments set: ```shell --set serviceNameOverride="-nginx-ingress" --set controller.name="" ``` It might look like this: ```shell helm upgrade test-release oci://ghcr.io/nginx/charts/nginx-ingress --version 0.19.0 --set controller.kind=deployment/daemonset --set controller.nginxplus=false/true --set controller.image.pullPolicy=Always --set serviceNameOverride="test-release-nginx-ingress" --set controller.name="" -f values.yaml ``` Once the upgrade process has finished, use `kubectl describe` on the deployment to verify the change by reviewing its events: ```shell Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 9m11s deployment-controller Scaled up replica set test-release-nginx-ingress- to 1 Normal ScalingReplicaSet 101s deployment-controller Scaled up replica set test-release-nginx-ingress- to 1 Normal ScalingReplicaSet 98s deployment-controller Scaled down replica set test-release-nginx-ingress- to 0 from 1 ``` ## Custom NGINX templates If you use custom NGINX configuration templates, review and apply any upstream template changes before upgrading. For more information, see [Custom templates](/nic/configuration/global-configuration/custom-templates.md). **warning:** In version 5.5, the default server block was removed from `main-template` and is now generated by `ingress-template`. You must update both templates before upgrading. If you had customizations inside the default server block in your `main-template`, the patch will remove them. Migrate those changes to your `ingress-template` manually. 1. Clone the repository: ```shell git clone https://github.com/nginx/kubernetes-ingress.git ``` 2. Generate a diff between the version you are upgrading from and the version you are upgrading to. For `main-template` and `ingress-template`, use `version1`. For `virtualserver-template` and `transportserver-template`, use `version2`: ```shell git diff v..v -- internal/configs/version1/ > upstream.patch git diff v..v -- internal/configs/version2/ > upstream.patch ``` 3. Extract your custom template from the ConfigMap to a local file. The valid key names are `main-template`, `ingress-template`, `virtualserver-template`, and `transportserver-template`: ```shell kubectl get configmap -n nginx-ingress -o jsonpath='{.data.}' > my-template.tmpl ``` 4. Apply the patch to your custom template: ```shell patch my-template.tmpl < upstream.patch ``` If the patch fails to apply some changes, review the rejected changes in `my-template.tmpl.rej` and apply them by manually. 5. Update the template key with the patched content. If you manage your ConfigMap directly, update the relevant key in your manifest and run `kubectl apply -f `. If you use Helm, set the key under `controller.config.entries` in your values file and run `helm upgrade`. -------------------------------------------------------------------------------- title: "Upgrade NGINX Ingress Controller LTS" url: https://docs.nginx.com/nginx-ingress-controller/lts/install/upgrade/index.md -------------------------------------------------------------------------------- This document describes how to upgrade F5 NGINX Ingress Controller LTS to a new patch release. Many of the nuances in upgrade paths relate to how custom resource definitions (CRDs) are managed. ## Upgrading to a new LTS patch release ### Upgrade NGINX Ingress Controller LTS CRDs To upgrade the CRDs, pull the Helm chart source, then use _kubectl apply_: ```shell helm pull oci://ghcr.io/nginx/charts/nginx-ingress-lts --untar --version kubectl apply -f crds/ ``` Alternatively, CRDs can be upgraded without pulling the chart by running: ```shell kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v/deploy/crds.yaml ``` In the above command, `v` represents the version of the NGINX Ingress Controller LTS release rather than the Helm chart version. **note:** The following warning is expected and can be ignored: `Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply`. Check the [release notes](https://www.github.com/nginx/kubernetes-ingress/releases) for a new release for any special upgrade procedures. ### Upgrade NGINX Ingress Controller LTS charts Once the CRDs have been upgraded, you can then upgrade the release chart. The command depends on if you installed the chart using the registry or from source. To upgrade a release named _my-release_, use the following command: #### OCI registry ```shell helm upgrade my-release oci://ghcr.io/nginx/charts/nginx-ingress-lts --version ``` #### Source ```shell helm upgrade my-release . ``` -------------------------------------------------------------------------------- title: "Migrate from Ingress-NGINX Controller to NGINX Ingress Controller" url: https://docs.nginx.com/nginx-ingress-controller/install/migrate-ingress-nginx/index.md -------------------------------------------------------------------------------- This document describes how to migrate from the community-maintained Ingress-NGINX Controller to F5 NGINX Ingress Controller. ## Overview This page explains two different ways to migrate from the community-maintained [Ingress-NGINX Controller](https://github.com/kubernetes/ingress-nginx) project to NGINX Ingress Controller: using NGINX's Ingress Resources or with Kubernetes's built-in Ingress Resources. This is typically because of implementation differences, and to take advantage of features such as [NGINX Plus integration](/nic/overview/nginx-plus). **Note:** For more information, you may want to read the two-part blog series from the [NGINX Community Blog](https://blog.nginx.org/) on Migrating from ingress-nginx to NGINX Ingress Controller: - [Part one](https://blog.nginx.org/blog/migrating-ingress-controllers-part-one) - [Part two](https://blog.nginx.org/blog/migrating-ingress-controllers-part-2) ## Before you begin To complete the instructions in this guide, you need the following: - A working knowledge of [Ingress Controllers](/glossary/#k8s-ingress-controller). - An [NGINX Ingress Controller installation](/nic/install/) on the same host as an existing Ingress-NGINX Controller. There are two primary paths for migrating between the community Ingress-NGINX Controller to NGINX Ingress Controller: - Using NGINX Ingress Resources - Using Kubernetes Ingress Resources. ## Migration with NGINX Ingress resources This path uses Kubernetes Ingress Resources to set root permissions, then NGINX Ingress Resources for configuration using custom resource definitions (CRDs): * [VirtualServer and VirtualServerRoute](/nic/configuration/virtualserver-and-virtualserverroute-resources) * [TransportServer](/nic/configuration/transportserver-resource) * [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource) * [Policy](/nic/configuration/policy-resource) ### Configuring SSL termination and HTTP path-based routing The following two code examples correspond to a Kubernetes Ingress Resource and an [NGINX VirtualServer Resource](/nic/configuration/virtualserver-and-virtualserverroute-resources). Although the syntax and indentation is different, they accomplish the same basic Ingress functions, used for SSL termination and Layer 7 path-based routing. Kubernetes Ingress Resource: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-test spec: tls: - hosts: - foo.bar.com secretName: tls-secret rules: - host: foo.bar.com http: paths: - path: /login backend: serviceName: login-svc servicePort: 80 - path: /billing serviceName: billing-svc servicePort: 80 ``` NGINX VirtualServer Resource: ```yaml apiVersion: networking.k8s.io/v1 kind: VirtualServer metadata: name: nginx-test spec: host: foo.bar.com tls: secret: tls-secret upstreams: - name: login service: login-svc port: 80 - name: billing service: billing-svc port: 80 routes: - path: /login action: pass: login - path: /billing action: pass: billing ``` ### Configuring TCP/UDP load balancing and TLS passthrough NGINX Ingress Controller exposes TCP and UDP services using [TransportServer](/nic/configuration/transportserver-resource) and [GlobalConfiguration](/nic/configuration/global-configuration/globalconfiguration-resource) resources. These resources provide a broad range of options for TCP/UDP and TLS Passthrough load balancing. By contrast, the community Ingress-NGINX Controller exposes TCP/UDP services by using a Kubernetes ConfigMap object. ### Convert Ingress-NGINX Controller annotations to NGINX Ingress resources Kubernetes deployments often need to extend basic Ingress rules for advanced use cases such as canary and blue-green deployments, traffic throttling, and ingress-egress traffic manipulation. The community Ingress-NGINX Controller implements many of these using Kubernetes annotations with custom Lua extensions. These custom Lua extensions are intended for specific NGINX Ingress resource definitions and may not be as granular as required for advanced use cases. The following examples show how to convert these annotations into NGINX Ingress Controller Resources. #### Canary deployments Canary and blue-green deployments allow you to push code changes to production environments without disrupting existing users. NGINX Ingress Controller runs them on the data plane: to migrate from the community Ingress-NGINX Controller, you must map the latter's annotations to [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources). The Ingress-NGINX Controller evaluates canary annotations in the following order: 1. _nginx.ingress.kubernetes.io/canary-by-header_ 1. _nginx.ingress.kubernetes.io/canary-by-cookie_ 1. _nginx.ingress.kubernetes.io/canary-by-weight_ For NGINX Ingress Controller to evalute them the same way, they must appear in the same order in the VirtualServer or VirtualServerRoute Manifest. Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "httpHeader" ``` NGINX Ingress Controller: ```yaml matches: - conditions: - header: httpHeader value: never action: pass: echo - header: httpHeader value: always action: pass: echo-canary action: pass: echo ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-header: "httpHeader" nginx.ingress.kubernetes.io/canary-by-header-value: "my-value" ``` NGINX Ingress Controller: ```yaml matches: - conditions: - header: httpHeader value: my-value action: pass: echo-canary action: pass: echo ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-by-cookie: "cookieName" ``` NGINX Ingress Controller: ```yaml matches: - conditions: - cookie: cookieName value: never action: pass: echo - cookie: cookieName value: always action: pass: echo-canary action: pass: echo ``` #### Traffic control Environments using microservices tend to use extensive traffic-control policies to manage ephemeral applications using circuit breaking and rate and connection limiting to prevent error conditions due to unhealthy states or abnormal behavior. The following examples map Ingress-NGINX Controller annotations to NGINX [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources) for rate limiting, custom HTTP errors, custom default backend and URI rewriting. Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/custom-http-errors: "code" nginx.ingress.kubernetes.io/default-backend: "default-svc" ``` NGINX Ingress Controller: ```yaml errorPages: - codes: [code] redirect: code: 301 url: default-svc ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/limit-connections: "number" ``` NGINX Ingress Controller: ```yaml http-snippets: | limit_conn_zone $binary_remote_addr zone=zone_name:size; routes: - path: /path location-snippets: | limit_conn zone_name number; ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/limit-rate: "number" nginx.ingress.kubernetes.io/limit-rate-after: "number" ``` NGINX Ingress Controller: ```yaml location-snippets: | limit_rate number; limit_rate_after number; ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/limit-rpm: "number" nginx.ingress.kubernetes.io/limit-burst-multiplier: "multiplier" ``` NGINX Ingress Controller: ```yaml rateLimit: rate: number/m burst: number * multiplier key: ${binary_remote_addr} zoneSize: size ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/limit-rps: "number" nginx.ingress.kubernetes.io/limit-burst-multiplier: "multiplier" ``` NGINX Ingress Controller: ```yaml rateLimit: rate: number/s burst: number * multiplier key: ${binary_remote_addr} zoneSize: size ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/limit-whitelist: "CIDR" ``` NGINX Ingress Controller: ```yaml http-snippets: | server-snippets: | ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/rewrite-target: "URI" ``` NGINX Ingress Controller: ```yaml rewritePath: "URI" ``` There are four Ingress-NGINX Controller annotations without NGINX Ingress resource fields yet: they must be handled using snippets. - _nginx.ingress.kubernetes.io/limit-connections_ - _nginx.ingress.kubernetes.io/limit-rate_ - _nginx.ingress.kubernetes.io/limit-rate-after_ - _nginx.ingress.kubernetes.io/limit-whitelist_ #### Header manipulation Manipulating HTTP headers is useful in many cases, as they contain information that is important and relevant to systems involved in HTTP transactions. The community Ingress-NGINX Controller supports enabling and configuring cross-origin resource sharing (CORS) headings used by AJAX applications, where front-end Javascript code interacts with backend applications or web servers. These code blocks show how the Ingress-NGINX annotations correspond to NGINX Ingress Controller [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources). Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/enable-cors: "true" nginx.ingress.kubernetes.io/cors-allow-credentials: "true" nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For" nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS" nginx.ingress.kubernetes.io/cors-allow-origin: "*" nginx.ingress.kubernetes.io/cors-max-age: "seconds" ``` NGINX Ingress Controller: ```yaml responseHeaders: add: - name: Access-Control-Allow-Credentials value: "true" - name: Access-Control-Allow-Headers value: "X-Forwarded-For" - name: Access-Control-Allow-Methods value: "PUT, GET, POST, OPTIONS" - name: Access-Control-Allow-Origin value: "*" - name: Access-Control-Max-Age value: "seconds" ``` #### Proxying and load balancing NGINX Ingress Controller has multiple proxy and load balancing functionalities you may want to configure based on the use case, such as configuring the load balancing algorithm and the timeout and buffering settings for proxied connections. This table shows how Ingress-NGINX Controller annotations map to statements in the upstream field for [VirtualServer and VirtualServerRoute resources](/nic/configuration/virtualserver-and-virtualserverroute-resources), covering load balancing, proxy timeout, proxy buffering and connection routing for a services' ClusterIP address and port. | Ingress-NGINX Controller | NGINX Ingress Controller | | ------------------------ | ------------------------ | | _nginx.ingress.kubernetes.io/load-balance_ | _lb-method_ | | _nginx.ingress.kubernetes.io/proxy-buffering_ | _buffering_ | | _nginx.ingress.kubernetes.io/proxy-buffers-number_ | _buffers_ | | _nginx.ingress.kubernetes.io/proxy-buffer-size_ | _buffers_ | | _nginx.ingress.kubernetes.io/proxy-connect-timeout_ | _connect-timeout_ | | _nginx.ingress.kubernetes.io/proxy-next-upstream_ | _next-upstream_ | | _nginx.ingress.kubernetes.io/proxy-next-upstream-timeout_ | _next-upstream-timeout_ | | _nginx.ingress.kubernetes.io/proxy-read-timeout_ | _read-timeout_ | | _nginx.ingress.kubernetes.io/proxy-send-timeout_ | _send-timeout_ | | _nginx.ingress.kubernetes.io/service-upstream_ | _use-cluster-ip_ | | _nginx.ingress.kubernetes.io/affinity_ (cookie) | _sticky-cookie-services_ | #### mTLS authentication mTLS authentication is a way of enforcing mutual authentication on traffic entering and exiting a cluster (north-sourth traffic). This secure form of communication is common within a service mesh, commonly used in strict zero-trust environments. NGINX Ingress Controller layer can handle mTLS authentication for end systems through the presentation of valid certificates for external connections. It accomplishes this through [Policy](/nic/configuration/policy-resource) resources, which correspond to Ingress-NGINX Controller annotations for [client certificate authentication](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-certificate-authentication) and [backend certificate authentication](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-certificate-authentication). Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/auth-tls-secret: secretName nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" ``` NGINX Ingress Controller: ```yaml ingressMTLS: clientCertSecret: secretName verifyClient: "on" verifyDepth: 1 ``` Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/proxy-ssl-secret: "secretName" nginx.ingress.kubernetes.io/proxy-ssl-verify: "on|off" nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "1" nginx.ingress.kubernetes.io/proxy-ssl-protocols: "TLSv1.2" nginx.ingress.kubernetes.io/proxy-ssl-ciphers: "DEFAULT" nginx.ingress.kubernetes.io/proxy-ssl-name: "server-name" nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on|off" ``` NGINX Ingress Controller: ```yaml egressMTLS: tlsSecret: secretName verifyServer: true|false verifyDepth: 1 protocols: TLSv1.2 ciphers: DEFAULT sslName: server-name serverName: true|false ``` #### Session persistence You can use [Policy](/nic/configuration/policy-resource) resources or annotations for session persistence, which have corresponding annotations for the community Ingress-NGINX Controller. Session persistence via sticky cookie is available in both NGINX (since version 1.29.6) and NGINX Plus. Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/affinity: "cookie" nginx.ingress.kubernetes.io/session-cookie-name: "cookieName" nginx.ingress.kubernetes.io/session-cookie-expires: "3600" nginx.ingress.kubernetes.io/session-cookie-path: "/route" nginx.ingress.kubernetes.io/session-cookie-secure: "true" ``` NGINX Ingress Controller: ```yaml sessionCookie: enable: true name: cookieName expires: 3h path: /route secure: true ``` ## Migration with Kubernetes Ingress resources The other option for migrating from the community Ingress-NGINX Controller to NGINX Ingress Controller is using only [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) and [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/) from standard Kubernetes resources, potentially relying on [mergeable Ingress types](https://github.com/nginx/kubernetes-ingress/tree/main/examples/ingress-resources/mergeable-ingress-types). This ensures that all configuration is kept in the Ingress object. **Note:** You should avoid altering the `spec` field of the Ingress resource when taking this option. Ingress-NGINX Controller and NGINX Ingress Controller differ slightly in their implementations: changing the Kubernetes Ingress can create incompatibility issues. ### Advanced configuration with annotations This table maps the Ingress-NGINX Controller annotations to NGINX Ingress Controller's equivalent annotations, and the respective NGINX Directive. | Ingress-NGINX Controller | NGINX Ingress Controller | NGINX Directive | | ------------------------ | ------------------------ | --------------- | | [_nginx.ingress.kubernetes.io/configuration-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#configuration-snippet) | [_nginx.org/location-snippets_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#snippets-and-custom-templates) | N/A | | [_nginx.ingress.kubernetes.io/load-balance_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-nginx-load-balancing) (1) | [_nginx.org/lb-method_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#backend-services-upstreams) | [_random two least_conn_](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#random) | | [_nginx.ingress.kubernetes.io/proxy-buffering_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffering) | [_nginx.org/proxy-buffering_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_buffering_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering) | | [_nginx.ingress.kubernetes.io/proxy-buffers-number_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffers-number) | [_nginx.org/proxy-buffers_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_buffers_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffers) | | [_nginx.ingress.kubernetes.io/proxy-buffer-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffer-size) | [_nginx.org/proxy-buffer-size_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_buffer_size_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) | | [_nginx.ingress.kubernetes.io/proxy-busy-buffers-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-busy-buffers-size) | [_nginx.org/proxy-busy-buffers-size_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_busy_buffers_size_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_busy_buffers_size) | | [_nginx.ingress.kubernetes.io/client-body-buffer-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-body-buffer-size) | [_nginx.org/client-body-buffer-size_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_client_body_buffer_size_](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) | | [_nginx.ingress.kubernetes.io/proxy-connect-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-connect-timeout_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_connect_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout) | | [_nginx.ingress.kubernetes.io/proxy-read-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-read-timeout_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_read_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) | | [_nginx.ingress.kubernetes.io/proxy-next-upstream_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-next-upstream_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_next_upstream_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) | | [_nginx.ingress.kubernetes.io/proxy-next-upstream-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-next-upstream-timeout_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_next_upstream_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) | | [_nginx.ingress.kubernetes.io/proxy-next-upstream-tries_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-next-upstream-tries_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_next_upstream_tries_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) | | [_nginx.ingress.kubernetes.io/proxy-send-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts) | [_nginx.org/proxy-send-timeout_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#general-customization) | [_proxy_send_timeout_](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout) | | [_nginx.ingress.kubernetes.io/rewrite-target_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rewrite) | [_nginx.org/rewrite-target_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#request-uriheader-manipulation) | [_rewrite_](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite) | | [_nginx.ingress.kubernetes.io/app-root_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rewrite) | [_nginx.org/app-root_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#request-uriheader-manipulation) | N/A | | [_nginx.ingress.kubernetes.io/ssl-ciphers_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#ssl-ciphers) | [_nginx.org/ssl-ciphers_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#auth-and-ssltls) | [_ssl_ciphers_](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) | | [_nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#ssl-ciphers) | [_nginx.org/ssl-prefer-server-ciphers_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#auth-and-ssltls) | [_ssl_prefer_server_ciphers_](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_prefer_server_ciphers) | | [_nginx.ingress.kubernetes.io/server-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-snippet)| [_nginx.org/server-snippets_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#snippets-and-custom-templates) | N/A | | [_nginx.ingress.kubernetes.io/ssl-redirect_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-side-https-enforcement-through-redirect) (2) | [_nginx.org/ssl-redirect_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#auth-and-ssltls) | N/A | | [_nginx.ingress.kubernetes.io/affinity_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#session-affinity)| [_nginx.org/sticky-cookie-services_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#backend-services-upstreams) | [_sticky_](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_cookie)| 1. Ingress-NGINX Controller implements some of its load balancing algorithms with Lua, which may not have an equivalent in NGINX Ingress Controller. 1. To redirect HTTP (80) traffic to HTTPS (443), NGINX Ingress Controller uses built-in NGINX `if` conditions while Ingress-NGINX Controller uses Lua. For [_nginx.ingress.kubernetes.io/force-ssl-redirect_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-side-https-enforcement-through-redirect) behavior (which works when SSL is terminated at an external load balancer), use [_nginx.org/redirect-to-https_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#auth-and-ssltls). The following two snippets outline Ingress-NGINX Controller annotations that correspond to annotations for NGINX Ingress Controller. Ingress-NGINX Controller: ```yaml nginx.ingress.kubernetes.io/affinity: "cookie" nginx.ingress.kubernetes.io/session-cookie-name: "cookie_name" nginx.ingress.kubernetes.io/session-cookie-expires: "seconds" nginx.ingress.kubernetes.io/session-cookie-path: "/route" ``` NGINX Ingress Controller: ```yaml nginx.org/sticky-cookie-services: "serviceName=example-svc cookie_name expires=time path=/route" ``` **Note:** NGINX Ingress Controller has additional annotations for features using NGINX Plus that have no Ingress-NGINX Controller equivalent, such as active health checks and authentication using JSON Web Tokens (JWTs). The following table outlines annotation conversions for enabling TLS and gRPC protocols to backend (upstream) services. | Ingress-NGINX Controller | NGINX Ingress Controller | | ------------------------ | ------------------------ | | [_nginx.ingress.kubernetes.io/backend-protocol_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol): "HTTPS" | [_nginx.org/ssl-services_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#backend-services-upstreams): "ssl_service_name" | | [_nginx.ingress.kubernetes.io/backend-protocol_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol): "GRPC" | [_nginx.org/grpc-services_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#backend-services-upstreams): "grpc_service_name" | | [_nginx.ingress.kubernetes.io/backend-protocol_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol): "GRPCS" | [_nginx.org/ssl-services_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#backend-services-upstreams): "grpc_service_name"
[_nginx.org/grpc-services_](/nic/configuration/ingress-resources/advanced-configuration-with-annotations.md#backend-services-upstreams): "grpc_service_name" | **Note:** GRPC services require Ingresses with TLS termination and HTTP/2 enabled (see [_http2 ConfigMap key_](/nic/configuration/global-configuration/configmap-resource.md#listeners)). ### Global configuration with ConfigMaps This table maps the Ingress-NGINX Controller ConfigMap keys to NGINX Ingress Controller's equivalent ConfigMap keys. | Ingress-NGINX Controller | NGINX Ingress Controller | | ------------------------ | ------------------------ | | [_disable-access-log_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#disable-access-log) | [_access-log-off_](/nic/configuration/global-configuration/configmap-resource.md#logging) | | [_error-log-level_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#error-log-level) | [_error-log-level_](/nic/configuration/global-configuration/configmap-resource.md#logging) | | [_hsts_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#hsts) | [_hsts_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_hsts-include-subdomains_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#hsts-include-subdomains) | [_hsts-include-subdomains_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_hsts-max-age_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#hsts-max-age) | [_hsts-max-age_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_http-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#http-snippet) | [_http-snippets_](/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) | | [_keep-alive_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#keep-alive) | [_keepalive-timeout_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_keep-alive-requests_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#keep-alive-requests) | [_keepalive-requests_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_load-balance_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#load-balance) | [_lb-method_](/nic/configuration/global-configuration/configmap-resource.md#backend-services-upstreams) | | [_location-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#location-snippet) | [_location-snippets_](/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) | | [_log-format-escape-json_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#log-format-escape-json) | [_log-format-escaping: "json"_](/nic/configuration/global-configuration/configmap-resource.md#logging) | | [_log-format-stream_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#log-format-stream) | [_stream-log-format_](/nic/configuration/global-configuration/configmap-resource.md#logging) | | [_log-format-upstream_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#log-format-upstream) | [_log-format_](/nic/configuration/global-configuration/configmap-resource.md#logging) | | [_main-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#main-snippet) | [_main-snippets_](/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) | | [_max-worker-connections_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-connections) | [_worker-connections_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_max-worker-open-files_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#max-worker-open-files) | [_worker-rlimit-nofile_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-body-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-body-size) | [_client-max-body-size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_client-body-buffer-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#client-body-buffer-size) | [_client-body-buffer-size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-buffering_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-buffering) | [_proxy-buffering_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-buffers-number_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-buffers-number) | [_proxy-buffers: number size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-buffer-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-buffer-size) | [_proxy-buffers: number size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-connect-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-connect-timeout) | [_proxy-connect-timeout_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-read-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-read-timeout) | [_proxy-read-timeout_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-send-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-send-timeout) | [_proxy-send-timeout_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_server-name-hash-bucket-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#server-name-hash-bucket-size) | [_server-names-hash-bucket-size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_proxy-headers-hash-max-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-headers-hash-max-size) | [_server-names-hash-max-size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_server-snippet_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#server-snippet) | [_server-snippets_](/nic/configuration/global-configuration/configmap-resource.md#snippets-and-custom-templates) | | [_server-tokens_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#server-tokens) | [_server-tokens_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_ssl-ciphers_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-ciphers) | [_ssl-ciphers_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_ssl-dh-param_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-dh-param) | [_ssl-dhparam-file_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_ssl-protocols_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-protocols) | [_ssl-protocols_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_ssl-redirect_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#ssl-redirect) | [_ssl-redirect_](/nic/configuration/global-configuration/configmap-resource.md#auth-and-ssltls) | | [_upstream-keepalive-connections_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#upstream-keepalive-connections) | [_keepalive_](/nic/configuration/global-configuration/configmap-resource.md#backend-services-upstreams) | | [_use-http2_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-http2) | [_http2_](/nic/configuration/global-configuration/configmap-resource.md#listeners) | | [_use-proxy-protocol_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol) | [_proxy-protocol_](/nic/configuration/global-configuration/configmap-resource.md#listeners) | | [_variables-hash-bucket-size_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#variables-hash-bucket-size) | [_variables-hash-bucket-size_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_worker-cpu-affinity_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-cpu-affinity) | [_worker-cpu-affinity_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_worker-processes_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-processes) | [_worker-processes_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | | [_worker-shutdown-timeout_](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#worker-shutdown-timeout) | [_worker-shutdown-timeout_](/nic/configuration/global-configuration/configmap-resource.md#general-customization) | -------------------------------------------------------------------------------- title: "Configure remote F5 WAF for NGINX bundle sources" url: https://docs.nginx.com/nginx-ingress-controller/integrations/app-protect-waf-v5/bundle-sources/index.md description: Configure NGINX Ingress Controller to fetch F5 WAF for NGINX policy bundles from NGINX One Console, NGINX Instance Manager, or an HTTPS server. -------------------------------------------------------------------------------- ## Overview This guide explains how to configure NGINX Ingress Controller to fetch pre-compiled F5 WAF for NGINX policy bundles from a remote source for VirtualServer resources, instead of manually copying bundles on to the cluster. This guide focuses on source-specific configuration details and validation steps. You can fetch bundles from: - **[NGINX One Console](/nginx-one-console/)** — for policies compiled and managed through NGINX One Console - **[NGINX Instance Manager](/nim/)** — for policies compiled and managed through NGINX Instance Manager - **[HTTPS](#https)** — for compiled `.tgz` bundles hosted on any HTTPS server Complete end-to-end NGINX Ingress Controller with F5 WAF for NGINX bundle source examples are available on GitHub: [N1C and NIM examples](https://github.com/nginx/kubernetes-ingress/tree/v/examples/custom-resources/waf-management-plane) and [HTTPS bundle server files](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/waf-bundle-server). ## NGINX One Console ### Before you begin - NGINX Ingress Controller deployed with [F5 WAF for NGINX v5](/nic/integrations/app-protect-waf-v5/installation.md). You can also [install with Helm](/nic/install/waf-helm.md). - An [NGINX One Console](/nginx-one-console/) account with a published WAF policy. See [Manage policies](/nginx-one-console/waf-integration/policy/_index.md). - A VirtualServer resource to attach the WAF policy to. **Note:** NGINX Ingress Controller does not trigger compilation. Compilation happens when a policy is published in NGINX One Console. Ensure the policy has been published and a compiled bundle is available before continuing. ### Create a credentials Secret Create a Secret of type `nginx.com/waf-bundle` in the same namespace as the Policy. The Secret must contain a `token` key with your NGINX One Console API token: To create an API token, see [Authentication](/nginx-one-console/api/authentication.md). ```shell kubectl create secret generic n1c-credentials \ --type=nginx.com/waf-bundle \ --from-literal=token= ``` ### Create a WAF Policy Create a Policy resource using `apBundleSource` with `type: N1C`: ```yaml apiVersion: k8s.nginx.org/v1 kind: Policy metadata: name: waf-policy spec: waf: enable: true apBundleSource: type: N1C url: "https://.console.ves.volterra.io" policyName: "my-blocking-policy" policyNamespace: "default" secret: "n1c-credentials" enablePolling: true pollInterval: "5m" ``` Replace `` with your NGINX One Console tenant hostname, `policyName` with the name of your published policy, and `policyNamespace` with the NGINX One Console namespace where the policy resides. **Note:** The field name is `policyName` for both `apBundleSource` and `apLogBundleSource`. In `apBundleSource`, set it to the published WAF policy name. In `apLogBundleSource`, set it to the log profile name (for example, `secops_dashboard`). **Note:** To skip TLS verification for testing, add `insecureSkipVerify: true` to the bundle source. Do not use this in production. ### Apply the policy to a VirtualServer After `waf-policy` is created, apply a VirtualServer that references it in `spec.policies`. ```yaml apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: webapp spec: host: webapp.example.com policies: - name: waf-policy upstreams: - name: webapp service: webapp-svc port: 80 routes: - path: / action: pass: webapp ``` For complete HTTPS setup manifests, see the [bundle server files](https://github.com/nginx/kubernetes-ingress/tree/v/examples/shared-examples/waf-bundle-server). ### Verify the bundle was fetched 1. Check the Policy events for a successful fetch: ```shell kubectl describe policy waf-policy ``` Look for a `Normal` event confirming the bundle was fetched. If you see a `Warning` event, check the message for the cause — common issues include an incorrect `policyName`, an invalid token, or a policy that has not been published yet. 1. Send a legitimate request to confirm traffic flows normally: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \ http://webapp.example.com:$IC_HTTP_PORT/ ``` 1. Send a malicious request to confirm WAF is blocking: ```shell curl --resolve webapp.example.com:$IC_HTTP_PORT:$IC_IP \ "http://webapp.example.com:$IC_HTTP_PORT/