End of Sale Notice:

F5 NGINX is announcing the End of Sale (EoS) for NGINX Controller API Management Module, effective January 1, 2024.

F5 maintains generous lifecycle policies that allow customers to continue support and receive product updates. Existing NGINX Controller API- Management customers can continue to use the product past the EoS date. License renewals are not available after September 30, 2024.

See our End of Sale announcement for more details.

End of Sale Notice:

F5 NGINX is announcing the End of Sale (EoS) for NGINX Controller Application Delivery Module, effective January 1, 2024.

F5 maintains generous lifecycle policies that allow customers to continue support and receive product updates. Existing NGINX Controller Application Delivery customers can continue to use the product past the EoS date. License renewals are not available after September 30, 2024.

See our End of Sale announcement for more details.

NGINX Controller v3.0.0–3.18.3

Learn about the new features, enhancements, resolved issues, and known issues in F5 NGINX Controller.

Release Notes

Here you can find the release information for F5 NGINX Controller v3. NGINX Controller is NGINX’s control-plane solution that manages the NGINX data plane. Built on a modular architecture, NGINX Controller enables you to manage the entire lifecycle of NGINX Plus, whether it’s deployed as a load balancer, API gateway, or a proxy in a service mesh environment.

We encourage you to install the latest version of NGINX Controller to take advantage of the newest features and updates.

Technical support is provided for earlier versions of NGINX Controller that were released within two years of the current release.

See Also:

For related installation documentation, refer to the following publications:

NGINX Controller Installation Guide

NGINX Controller Technical Specifications Guide

November 23, 2021

NGINX Controller v3.18.3

Updates

Fixes a rare timeout issue that affected some customers.
Performance improvements.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R24
NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

July 26, 2021

NGINX Controller v3.18.2

Resolved Issues

Resolves an issue that created duplicate health checks for app components

A health check is generated for each component that has monitoring enabled. If a backend workload is shared with multiple components, and monitoring is the same for two or more of those components, then only one health check is generated now. This is to avoid duplicating health checks.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R24
NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

July 19, 2021

NGINX Controller v3.18.1

These release notes provide general information and describe known issues for NGINX Controller v3.18.1, in the following categories:

Resolved Issues
Supported Versions

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

The peer count alert needs to be configured with a multiplier of the reporting period (26507)
Enabling WAF when creating or updating a component using the web interface might fail for non-admin users (26579)

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R24
NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.18.0

June 15, 2021

NGINX Controller v3.18.0 includes the following new features, improvements, known issues, and bug fixes:

Updates
Vulnerability Fixes
New Issues
Supported Versions

Updates

Bug fixes and improvements.
Improvements to the Data Explorer make it easier to see your data’s dimensions and preview the discrete values.
Adds support for NGINX Plus R23 p1 and R24 p1.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action to take, see the linked AskF5 Solution Article for details.

Intra-cluster communication does not use TLS (5347)

Intra-cluster communication does not use TLS. The services within the NGINX Controller namespace are using cleartext protocols inside the cluster.

Solution Article: K97002210 | CVE-2021-23018
Agent configuration is world-readable (6580)

The agent configuration file /etc/controller-agent/agent.conf is world-readable with current permission bits set to 644.

Solution Article: K36926027 | CVE-2021-23021
API Key generated with weak cryptography (8867)

The NAAS API keys were generated using an insecure pseudo-random string and hashing algorithm that could lead to predictable keys.

Solution Article: K45263486 | CVE-2021-23020
The NGINX Controller admin password disclosed in supportpkg systemd logs (22633)

The NGINX Controller Administrator password may be exposed in the systemd.txt file included in the NGINX support package.

Solution Article: K04884013 | CVE-2021-23019

New Issues

Alerts

The peer count alert needs to be configured with a multiplier of the reporting period (26507)

The plus.upstream.peer.count metric, when used as a basis of an alert, reports the total number of healthy upstream hosts in a deployment. The alert for this metric incorrectly uses a SUM aggregation method.

Workaround:

The threshold value must be configured with a multiplier of the number of minutes in the reporting period. If your alert is set to a 2m reporting period, for example, and you have two configured upstreams, your threshold should be set to 4, not 2. If your reporting period is 10m, your threshold should be set to 20, not 2.

App Security

Enabling WAF when creating or updating a component using the web interface might fail for non-admin users (26579)

Using the web interface to create or update an app component with WAF might fail for non-admin users. An error similar to the following example is displayed: “Error updating component comp1: API Error (1235): User is not permitted to access this resource. Contact the system administrator.” Workaround:

Use the NGINX Controller REST API instead of the web interface, or upgrade to NGINX Controller v3.18.1 or later.

Documentation

Instance Groups API is not specified as a beta feature (25924)

The Instance Groups API was introduced as an experimental feature in NGINX Controller v3.17. The description does not explicitly state that the endpoints are in beta status. The Instance Groups API is a beta feature in both NGINX Controller v3.17 and v3.18 and is not recommended for use in production environments.

The workloadGroup section for the Components API is missing in the NGINX Controller REST API guide (26550)

In the NGINX Controller REST API guide, the workloadGroup section for the Components API is missing in both the Application Delivery and API Management specs.

Workaround:

Refer to the following OpenAPI Specification for the workloadGroup. There are two types of workloadGroup: {WebWorkloadGroup and TcpUdpWorkloadGroup}. Each type references a WorkloadGroupCommon, and then follows with its specific objects.

WebWorkloadGroup:
      description: Group of servers hosting a part of a Web application represented by a Component.
      allOf:
        - $ref: '#/components/schemas/WorkloadGroupCommon' # Settings common to Web and TCP/UDP workloadGroups.
        - type: object
          properties:
            proxy:
              $ref: '#/components/schemas/WebProxy' # Proxy retry and timeout settings applicable to servers in a Web workloadGroup.
            sessionPersistence:
              $ref: '#/components/schemas/SessionPersistence' # SessionPersistence settings in a Web workloadGroup.
            uris:
              type: object
              description: |
                The URI for a server hosting a part of a Web application.
                It must conform to the format `schema://address[:port]`
                where schema is chosen from http or https, address is IP or hostname,
                schema and address must be provided.
                For example:
                - `http://192.0.2.247`
                - `https://192.0.2.247:8443`
                - `https://www.f5workload.com`
              additionalProperties:
                $ref: '#/components/schemas/WorkloadUri'
TcpUdpWorkloadGroup:
      description: Group of servers hosting a part of a TCP/UDP application represented by a Component.
      allOf:
        - $ref: '#/components/schemas/WorkloadGroupCommon' # Settings common to Web and TCP/UDP workloadGroups.
        - type: object
          properties:
            proxy:
              $ref: '#/components/schemas/TcpUdpProxy' # Proxy retry and timeout settings applicable to servers in a TcpUdp workloadGroup.
            uris:
              type: object
              description: |
                The URI for a server hosting a part of a TCP/UDP application.
                The URI must conform to the format `schema://address:port`
                where schema is chosen from tcp, udp, or tcp+tls, address is IP or hostname.
                All three of schema, address, and port must be provided.
                For example:
                - `tcp://192.0.2.247:8443`
                - `tcp+tls://192.0.2.247:8449`
                - `udp://www.f5workload.com:989`
              additionalProperties:
                $ref: '#/components/schemas/WorkloadUri'
    WorkloadGroupCommon:
      description: Settings common to Web and TCP/UDP workloadGroups.
      type: object
      properties:
        locationRefs:
          type: array
          items:
            $ref: 'https://gitlab.com/f5/nginx/controller/product/api-common/raw/master/schema/resource-common.yaml#/components/schemas/ResourceRef'
        loadBalancingMethod:
          $ref: '#/components/schemas/LoadBalancingMethod'
        dnsServiceDiscovery:
          $ref: '#/components/schemas/DNSServiceDiscovery'
    WorkloadUri:
      type: object
      properties:
        weight:
          type: integer
          minimum: 1
          default: 1
        maxConns:
          type: integer
          minimum: 0
          default: 0
        maxFails:
          type: integer
          minimum: 0
          default: 1
        failTimeout:
          x-f5-experimental: true
          type: string
          default: 10s
          pattern: '^[0-9]+[h|m|s]{1}$'
        isBackup:
          type: boolean
          default: false
        isDown:
          type: boolean
          default: false
        route:
          x-f5-experimental: true
          type: string
        srvService:
          type: string
        slowStart:
          x-f5-experimental: true
          type: integer
          minimum: 0
          default: 0
        isDrain:
          type: boolean
          default: false

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R24
NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.17.0

May 20, 2021

These release notes provide general information and describe known issues for NGINX Controller v3.17, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

Adds support for NGINX Plus R24

NGINX Controller v3.17 fully supports NGINX Plus R24 for configuration management and monitoring.
Adds support for NGINX App Protect 3.2

NGINX Controller v3.17 fully supports NGINX App Protect 3.2.
(Technical preview) Adds Debian 10 support

NGINX Controller Agent v3.17 and App Security Add-on v3.17 support Debian 10 as a technical preview. Functionality has been verified; however more testing is needed to finalize this feature. Look for updates in future releases. See the NGINX Plus Admin Guide for requirements for Debian 10.
(Beta) Bring your own custom NGINX App Protect Policy to configure WAF

Now, you can bring your own custom NGINX App Protect JSON declarative policy to use as your WAF policy with NGINX Controller, in addition to using the default policy. F5 Advanced WAF and BIG-IP Application Security Module (ASM) customers can convert their standardized WAF policy to an App Protect policy to use with NGINX Controller. Note: This feature is available only through the NGINX Controller API for this beta release.
The new Data Explorer view unlocks an easier way to view metrics

The new Data Explorer view gets you to your data faster, so you can understand what’s happening in your environment with ease. Effortlessly switch between contexts, metrics, and dimensions; specify a time range; set the aggregation mode; and more.
Improved user experience working with APIs

Working with APIs is easier now. We simplified navigating API definitions and related objects like versions and published APIs.
Easily enable or disable High Availability mode by updating gateways

Prior to NGINX Controller v3.17, you had to delete and recreate gateways to enable or disable HA (high availability) mode . Now, you can enable or disable HA mode on the fly by simply updating the existing gateway.
Adds capability to view the total number of Security Events matching dimensional filters

You can now view the total number of WAF violations when filtering Security Events by selected dimensions for more efficient analysis.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Installing NGINX Controller on a misconfigured NFS server fails with the error “Bundled DB did not start successfully” (22751)
TCP/UPD traffic misreported as “Web (HTTP) Components” after upgrading to NGINX Controller v3.15 (22866)
Update script reports “controller-postgres volume not found” if NGINX Controller uses external config database (23739)

New Issues

Red Hat Enterprise Linux 8 requires Xerces-C package to be installed before installing Controller Agent (25079)

When installing the NGINX Controller Agent on Red Hat Enterprise Linux 8, the metrics module fails to install if the Xerces-C library package hasn’t been installed already. An error similar to the following is returned:
```
Error:
Problem: conflicting requests
  - nothing provides libxerces-c-3.2.so()(64bit) needed by avrd-0.21.2_297315219.release_3_17-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

13. Installing metrics packages ... failed.
```
Despite this error, the Controller Agent installs successfully; however, application-centric metrics won’t be sent to NGINX Controller without this metrics module.

Workaround:

To resolve this issue, run the following commands to install Xerces-C from the EPEL Fedora repo:
```
wget https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/x/xerces-c-3.2.2-3.el8.x86_64.rpm
sudo rpm -Uvh xerces-c-3.2.2-3.el8.x86_64.rpm
```
You can perform this workaround before installing the Controller Agent package. If you encounter the error while installing the Controller Agent, try installing Xerces-C and then re-run the Controller Agent installation.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R24
NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.16.1

April 19, 2021

These release notes provide general information and describe known issues for NGINX Controller v3.16.1, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller with App Security Add-on supports NGINX App Protect 3.1

The App Security add-on now supports NGINX App Protect 3.1 on NGINX Plus R23.
NGINX Controller with App Security Add-on supports Ubuntu 20.04 LTS on the data plane

The App Security add-on now supports Ubuntu 20.04 LTS on the data plane using the Controller Agent and NGINX App Protect.
Association Token licenses no longer require port 8883

You no longer need to open port 8883 to license NGINX Controller with an Association Token. You still need to open port 443, however.

See Licensing NGINX Controller for instructions.
Adds support for SSL settings in health checks when multiple virtual hosts are defined per workload group member

Auto-generated health check location blocks now include any proxy_ssl settings that have been defined in the TLS object in the web backend object. For example, using the NGINX Controller REST API to set
```
"tls":{"isServerNameEnabled": true, "name": "$host"}
```
allows data plane traffic to have Server Name Indication (SNI) information that can be used where multiple virtual hosts exist per workload group member. Health checking also inherits these settings, so the SNI information is provided in the health checking context.
Active Directory integration adds support for LDAPS and unencrypted LDAP

Active Directory integration with NGINX Controller now supports LDAPS and unencrypted LDAP, in addition to StartTLS.

Refer to Configure Active Directory Integration for instructions on how to add an Active Directory provider to NGINX Controller for external authentication.
Data Forwarders are no longer restricted

While in beta, Data Forwarders had restrictions to limit the number of stream sources that could be defined. These restrictions have been lifted.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Links to API endpoints in the NGINX Controller API Reference docs don’t work (22415)
TCP/UDP component associated with multiple gateways on an instance causes config to fail to load (22850)

New Issues

The following issues are known to be present in this release. Look for updates to these issues in future NGINX Controller release notes.

Installation and Upgrade

Update script reports “controller-postgres volume not found” if NGINX Controller uses external config database (23739)

You may see an error similar to the following when updating NGINX Controller if you use an external config database:
```
Error from server (NotFound): persistentvolumes "controller-postgres" not found.
```
This error relates to an internal test being conducted by the update script and doesn’t affect the upgrade’s success. You can safely ignore this error.
Harmless errors reported when joining a node to a multi-node resilient cluster (23789)

When joining a control plane node to a multi-node resilient cluster, errors similar to the following may be displayed:
```
Error from server (NotFound): pods "kube-controller-manager-<host>" not found
```
```
Error from server (NotFound): pods "kube-scheduler-<host>" not found
```
These errors do not prevent the node from joining the cluster and can safely be ignored.
Upgrading NGINX Controller Agent fails with error “can’t find %s” (24019)

In rare cases, upgrading the NGINX Controller Agent may fail with the error “can’t find %s”.

If you look up the version for the ngninx-controller-agent service using apt or yum, you may see the version is 999.

Workaround:

To resolve this error, you’ll need to uninstall and then re-install the Controller Agent. Refer to the NGINX Controller Agent Installation Guide for instructions.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.15.0

March 11, 2021

These release notes provide general information and describe known issues for NGINX Controller v3.15.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

Analytics now fully supports containers

Containers are now fully supported! You might notice some differences in the analytics for containers vs. full hosts. This is because containers are unable to report the same metrics.

To get started using NGINX Controller with containers, see the docker-nginx-controller README on GitHub.
Introduces new metrics and dimensions for TCP/UDP
- NGINX Controller 3.15 adds four new metrics: bytes_sent, bytes_rcvd, connection_count, and connection_duration.
  - bytes_sent and bytes_rcvd are applicable to both TCP/UDP and HTTP; without any filtering, these metrics give aggregate values across HTTP and TCP/UDP.
  - connection_count and connection_duration are TCP specific.
- We’ve also added three new metric dimensions: proxied_protocol, family, and upstream_addr.
- For HTTP, metrics are collected at the end of a request/response interaction. For TCP/UDP, metrics are collected at the end of a connection setup/teardown interaction.
- These new metrics will show data that’s generated only after an upgrade to NGINX Controller v3.15. To view the metrics, you need to:
  - Upgrade NGINX Controller
  - Upgrade the Controller Agent
  - Republish all TCP/UDP components using the API or browser interface. Re-publishing is not required for HTTP gateways or components.
- For UDP, connection_duration will be the NGINX default proxy_timeout value (10 minutes), except if an “ICMP destination unreachable” message is received.
Introduces NGINX Controller’s own Dev Portal as the user interface for the NGINX Controller API Reference guide

The NGINX Controller docs are now using our own Dev Portal user interface to display the NGINX Controller API documentation. Check it out at:

https://docs.nginx.com/nginx-controller/api/reference/ctlr-v1/

Refer to the API Management documentation to learn more about using Dev Portals for your APIs.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

NGINX Controller update script reports a permission denied error when loading CA certificate (21211)
Agent doesn’t stash auxfiles in Docker container (21711)
Agent can’t marshal update payload in Docker container (21809)
Custom dashboard filters are cleared when edited, need to be reapplied (21922)

New Issues

The following issues are known to be present in this release. Look for updates to these issues in future NGINX Controller release notes.

Installation and Upgrade

Installing NGINX Controller on a misconfigured NFS server fails with the error “Bundled DB did not start successfully” (22751)

When installing NGINX Controller on an NFSv3 server, if the NFS requirements are not met, the installation may fail with an error similar to the following: “Bundled DB did not start successfully.”

Workaround:

See the NGINX Controller Technical Specifications Guide for the NFS requirements. The no_root_squash option must be set for the mount point on the NFS server. If this is not allowed, the owner of the path used for the analytics database must be set to 101:101, and the owner of the path for the config database must be set to 70:70.

After editing the exports file, the directories need to be removed to ensure no files are leftover from the previous installation. Then, enable the rpc-statd service and restart the NFS server.

Apps and Services

connect_timeout and send_timeout only take seconds as a specified value (21686)

If connect_timeout and send_timeout are specified in hours:minutes:seconds, the system returns an error response.

Workaround:

Only use seconds for these fields.
TCP/UDP component associated with multiple gateways on an instance causes config to fail to load (22850)

If a TCP/UDP component is associated with multiple gateways on the same instance – for example, an association of a unique listen IP per gateway placement, with a wildcard address defined in the URI within the component – the configuration fails to load.
TCP/UPD traffic misreported as “Web (HTTP) Components” after upgrading to NGINX Controller v3.15 (22866)

On non-Ubuntu systems, after upgrading to NGINX Controller v3.15, the TCP/UDP component statistics are not reported correctly. TCP/UDP traffic may be misattributed as “Web (HTTP) Components” requests.

Workaround:

After upgrading to v3.15, in addition to republishing any TCP/UDP components, you must restart the nginx and controller-agent services on any data paths that serve as a gateway for defined TCP or UDP components.

Complete the following steps in sequence:
1. Stop the nginx service:
```
systemctl stop nginx
```
2. Stop the controller-agent service:
```
systemctl stop controller-agent
```
3. Start the nginx service:
```
systemctl start nginx
```
4. Start the controller-agent service:
```
systemctl start controller-agent
```
After stopping and restarting the services, your connection count (TCP), connection duration (TCP), and bytes in/bytes out (TCP/UDP) will be reported correctly.

Infrastructure

Ubuntu and Debian Docker hosts may need cgroup swap limit enabled (19514)

When running a containerized instance that includes the NGINX Controller Agent on an Ubuntu or Debian Docker host, you may need to enable cgroup swap limit capabilities so that the Controller Agent can report swap metrics for those instances. For instructions, refer to the Docker post-installation topic “Your kernel does not support cgroup swap limit capabilities.”

Documentation

The httpHttpsRedirect property is not tagged as experimental in the API Reference (22561)

The httpHttpsRedirect property, which is part of the Components > Programmability schema for the NGINX Controller Application Delivery module, is currently experimental. The x-f5-experimental extension is, incorrectly, not displayed in the API Reference documentation for the property. Users should take note that this feature is not currently supported and may change in future versions with no guarantee of backward compatibility.
Links to API endpoints in the NGINX Controller API Reference docs don’t work (22415)

In the new NGINX Controller API Reference documentation, using a direct URL to an API endpoint – such as https://docs.nginx.com/nginx-controller/api/reference/ctlr-v1/services/docs/devPortals/v1/paths – will return a 404 error. Once you’ve received a 404 error from a direct endpoint URL, navigating by using the back and forward browser buttons will also produce 404 errors.

Workaround:

To access information about the NGINX Controller REST API endpoints, navigate to the desired endpoint by using the browser interface instead of direct links.
The Catalogs and Dev Portals API docs overview pages return 404 errors in the NGINX Controller browser interface (22942)

In the NGINX Controller browser interface, certain API Reference docs pages are returning 404 errors. These include the “overview” and “all endpoints” pages for the Catalogs and Dev Portals APIs.

Workaround:

You can access the Catalogs and Dev Portals API overview pages in the NGINX Controller docs online. To do so, go to https://docs.nginx.com/nginx-controller/api/reference/ctlr-v1 and navigate to the overview page for the desired endpoint.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.14.0

February 10, 2021

These release notes provide general information and describe known issues for NGINX Controller v3.14.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

Monitor your apps’ metrics with Alert Rules

Now you can create Alert Rules for app-centric metrics using the NGINX Controller web interface or REST API. Also, the web interface for Alerts has a new and improved design.

To create Alert Rules, refer to the Manage Alerts topic. For information about the metrics that you can monitor, see the NGINX Controller Metrics Catalog.
NGINX Controller with App Security Add-on supports NGINX App Protect 3.0

The App Security add-on now supports NGINX App Protect 3.0 on NGINX Plus R23.
View the percentage of requests that WAF has blocked or flagged Now, the Security Analytics graphs show the percentage of requests that have been blocked or flagged by the Web Application Firewall (WAF). This allows you to identify spikes or abnormalities based on the proportion of traffic rejected or flagged traffic by the WAF even as traffic increases.

Go to View App Security Analytics to get started.
Upgrades to health checking for web apps

Health checking for web apps has been extended as follows:
- A health check is sent for each virtual host on a workload group member/upstream member that is being monitored.
- The host header in the health check is set to the server name’s value in the server block from which the health check originated.
- All virtual hosts on a particular workload group member/upstream member will inherit any failures on any other virtual host on that member due to the present shared state implementation. This will be addressed in future releases.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Floating IP has to be removed to be updated (19011)
Multi-node, resilient deployments and automated backups of embedded config DB are not supported on AWS (20052)
NGINX Controller displays “Application Error” when new licenses with additional features are applied (20868)
NGINX Controller deletes some alert rules on upgrade to v3.13 (20882)
NGINX Controller may crash if control plane nodes are running different versions of NGINX Controller (21150)
Alerts for controller.agent.status are inconsistent and unreliable (21276)

New Issues

The following issues are known to be present in this release. Look for updates to these issues in future NGINX Controller release notes.

Backup and Restore

Restoring NGINX Controller from a backup without restoring the config and encryption keys causes the system to become unresponsive (22066)

When restoring NGINX Controller from a backup, you also need to restore the NGINX config and encryption keys; otherwise, the system may become unresponsive. For instructions, refer to Back Up & Restore Cluster Config and Encryption Keys.

Analytics

Custom dashboard filters are cleared when edited, need to be reapplied (21922)

When editing a custom dashboard that has a filter, the filter settings are cleared. If you save the custom dashboard before reapplying the filter settings, the saved filter settings may be lost.

Workaround:

If you need to edit a custom dashboard with a filter, you should reapply any cleared filter settings before saving the dashboard.

Infrastructure

NGINX Controller Agent doesn’t stash auxfiles in Docker container (21711)

In some cases, when running the NGINX Controller Agent in a Docker container, the Agent may fail to stash the auxfiles directory during a configuration change. If this happens, it may not be possible to roll back the update if the config change is unsuccessful.
NGINX Controller Agent can’t marshal update payload in Docker container (21809)

In some cases, when running in a Docker container, the NGINX Controller Agent may fail to marshal an update payload. This can affect metrics and alerts.

Licensing

License Usage column does not appear in the NGINX Controller web interface after restarting platform-mgr pod (20907)

When the platform-mgr pod is restarted, the Usage column on the Platform > License page may not show usage data for a brief time.

Workaround:

Wait at least 30 seconds, then refresh the webpage. The usage data will display if there’s any to data show.

Platform

Failovers on resilient NGINX Controller clusters on AWS can take up to 5 minutes (21586)

The automated failover from a failed control node to a working node can take up to 5 minutes on resilient NGINX Controller clusters on Amazon Web Services. During the failover, NGINX Controller may be unavailable while services are migrated and restarted.

Documentation

Steps to restore embedded config database from a specific backup file are incorrect (22044)

The steps to restore the embedded config database from a specific backup file, found in the user documentation that’s installed with NGINX Controller, are incorrect.

Workaround:

To restore the embedded config database, follow the instructions in the Backup & Restore Embedded Config Database guide that’s online.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.13.0

January 20, 2021

These release notes provide general information and describe known issues for NGINX Controller v3.13.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

Adds support for NGINX Streams (TCP & UDP)

NGINX Controller now supports TCP/UDP load balancing and routing along with the coincident support for data path HA for these protocols. TCP/UDP does not support metrics at this time – look for metrics support coming soon.

To learn more, check out the Manage Gateways and About Application Delivery topics.
Improved experience deploying the NGINX Controller with App Security Add-on

You no longer need to add security = True in the agent.conf file to get Security Events (WAF violation events) after installing the NGINX App Protect module on the data planes – NGINX Controller takes care of the configuration for you. An Agent restart is still needed.

To begin using NGINX Controller with the Application Security Add-on, follow the steps in the Trial NGINX Controller with App Security quick start guide.
NGINX Controller with App Security Add-on supports Ubuntu 18.04 LTS on the data plane

The App Security add-on now supports Ubuntu 18.04 LTS on the data plane using the Controller Agent and NGINX App Protect.
NGINX Controller with App Security Add-on supports NGINX App Protect 2.3

The App Security add-on now supports NGINX App Protect 2.3 on NGINX Plus R23.
Data forwarding for Splunk now includes NGINX Controller events

You can now forward all NGINX Controller events – including system and security events – to your Splunk HTTP Event Collector(s), similar to log forwarding.

To get started, refer to Forward Analytics Data to Splunk.
The data forwarder output format SPLUNK_HEC was renamed to SPLUNK

The output format type SPLUNK_HEC has been shortened to SPLUNK to make it more intuitive.
Usage information is now reported with licensing information

Now when you use the NGINX Controller web interface or the REST API to view licensing information, you’ll also be able to view usage details to ensure compliance with your license.

For instructions on how to view and manage your licenses, see License NGINX Controller.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

The nginx.master.reloads metric increments and sums incorrectly (14533)
Gateway configuration stuck at Configuring if referenced instances aren’t running App Protect processes (19441)
APIM License limits may display incorrectly (19919)
NGINX Controller reports missing nginx.conf.bak file when reloading NGINX Plus configuration times out (20227)
API Reference does not specify that TCP/UDP App Components are experimental (20285)

New Issues

The following issues are known to be present in this release. Look for updates to these issues in future NGINX Controller release notes.

Installation and Upgrade

NGINX Controller may crash if control plane nodes are running different versions of NGINX Controller (21150)

When adding control plane nodes to the cluster, NGINX Controller may crash if different NGINX Controller versions are installed on the nodes. To add control plane nodes to the cluster, ensure you install the same NGINX Controller von all the nodes.

Workaround:

If you encounter this issue, contact NGINX Support for assistance.

Alerts

NGINX Controller web interface may not display events if an ad blocker is enabled (18841)

In some cases, the NGINX Controller web interface may not display analytics or security events if an ad blocker is enabled.

Workaround:

Turn off any ad blockers while using the NGINX Controller web interface.
NGINX Controller deletes some alert rules on upgrade to v3.13 (20882)

NGINX Controller v3.13 contains changes to the functionality of Alerts, including Alert Rules and Alert Notifications. Alert rules that use the following metric types will not be carried over into v3.13 on upgrade from previous versions:
Deprecated:

As of NGINX Controller v3.13, the following Alerts functionality is deprecated:
- The ServiceNow Alerts integration is no longer supported. Subscribers of any type other than email will be ignored. Alert rules which do not have at least one email subscriber will not be migrated.
- The alert rule operator = is no longer available. Any existing alert rules with operator: = will be ignored.
- Alert rules created in versions prior to 3.13 that use the RATE aggregation will be removed on upgrade. These rules must be re-created manually. Alert rules that use the following metrics are affected:
  - plus.stream.upstream.bytes_sent
  - plus.stream.upstream.bytes_rcvd
  - system.io.iops_w
  - system.io.kbs_w
  - system.io.iops_r
  - system.io.kbs_r
  - system.net.bytes_sent
  - system.net.bytes_rcvd
  - plus.stream.bytes_sent
  - plus.stream.bytes_rcvd
  - plus.upstream.bytes_sent
  - plus.upstream.bytes_rcvd
  - nginx.workers.io.kbs_r
  - nginx.workers.io.kbs_w
  - nginx.http.request.count
  - nginx.http.request.body_bytes_sent
  - nginx.http.request.bytes_sent
  - nginx.http.conn.accepted
  - nginx.http.conn.dropped
  - plus.http.request.bytes_sent
  - plus.http.request.bytes_rcvd
Additionally, there is a change to the summary email schedule. Ongoing alert emails will not be sent every hour anymore. The summary email will only contain alerts that have started or expired in the past hour.

Workaround:
- Make note of any alerts that use the metrics types listed above.
- Re-create the alert rules when the upgrade to v3.13 is complete.
Alerts for controller.agent.status are inconsistent and unreliable (21276)

The alerts for controller.agent.status are emitted inconsistently and are unreliable in NGINX Controller v3.13. We recommend that you do not configure alerts for controller.agent.status or that you mute these alerts if you’re upgrading to or installing NGINX Controller v3.13.

App Security

NGINX Controller may fail to add an instance to a gateway if multiple components associated with the gateway have unique sets of WAF signature overrides (20752)

In cases where multiple components have unique sets of WAF signature overrides, adding a new instance reference to a gateway may result in a timeout.

Workaround:

To add an instance reference to a gateway, take the following steps:
- For components that reference the default policy, remove all signature overrides from components associated with the gateway that’s being changed.
- For components that reference a custom policy [Beta], replace the custom policy with the default policy without any signature override.
- Add the instance to the gateway.
- For components that referenced the default policy, add the signature overrides back to the components.
- For components that previously referenced a custom policy [Beta], replace the default policy with the custom policy.
Configuration pushes may time out if components with security enabled are in an error state (20920)

If multiple components with security enabled and signature overrides are in an error state, pushing configuration changes for additional components may time out.

Workaround:

To push configuration changes, take the following steps:
1. Remove the signature overrides from the components.
2. Resolve the component errors one at a time, waiting for each error to resolve and the component state to become “configured.”
3. Restore the signature overrides for each component one at a time, waiting for the state for each component to become “configured.”

Licensing

NGINX Controller displays “Application Error” when new licenses with additional features are applied (20868)

Applying a new license with additional features to a previously licensed version of NGINX Controller causes the web interface to display the message “Application Error.”

Workaround:

You can safely ignore this message. Wait approximately 30 seconds, and then refresh the page; the message should disappear.

Platform

Users with an environment-specific role can’t create or update Components using the browser interface (22021)

Users who have an environment-specific role with read/write permissions may not be able to create or update Components using the NGINX Controller browser interface. The system displays an error similar to the following: “Error Fetching Component. API Error (403): unauthorized.”

Workaround:

Use the /platform/roles endpoint in the NGINX Controller REST API to add the following permissions to the custom role:
```
{
  "path": "/api/v1/services/errorsets",
  "access": "READ"
},
{
  "path": "/api/v1/security/identity-providers",
  "access": "READ"
}
```
The errorsets resource is a newly added resource that’s needed to create or modify Components. The identity-providers resource needs to have READ permission to work in the NGINX Controller browser interface.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.12.1

December 30, 2020

This hotfix release resolves the known issue affecting version 3.12 described below.

Upgrading to NGINX Controller v3.12 fails on systems that don’t meet the minimum CPU requirements (20509):

NGINX Controller requires an 8-core CPU @ 2.4 GHz or higher.

If you’ve installed NGINX Controller on a system that does not have an 8-core CPU @ 2.4 GHz or higher, the upgrade to 3.12 may fail.

Caution:

Installing or upgrading NGINX Controller on systems that do not meet the minimum hardware requirements may cause NGINX Controller to become unresponsive.

Refer to the NGINX Controller Tech Specs guide for hardware requirements and system recommendations.

Following the workaround below will restore functionality. However, running on a system that does not meet the documented minimum requirements is not supported or recommended.

It is strongly recommended that you upgrade your system to meet the minimum requirements (8-core CPU). Doing so will ensure a successful upgrade to version 3.12.

Workaround:

To restore a failed upgrade, take the steps below:

Stop NGINX Controller by using the helper script:

/opt/nginx-controller/helper.sh controller stop

Wait for 1-2 minutes for all services to stop. You can run the command below to check the status:
```
/opt/nginx-controller/helper.sh controller status
```

Run the update script:

/opt/nginx-controller/update.sh --passive

Start NGINX Controller by using the helper script:

/opt/nginx-controller/helper.sh controller start

Wait 2-3 minutes for all services to start. Then, log in to the NGINX Controller user interface to verify that system functionality is restored.

Note:
If you’re not able to confirm that functionality has been restored by following the above steps once, try repeating them. In some test cases, repeating the steps was necessary to complete the upgrade.

NGINX Controller v3.12.0

December 15, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.12.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.12.0 includes the following updates:

Deploy NGINX Controller as a resilient cluster on a private cloud

Now you can deploy NGINX Controller as a multi-node cluster on a private cloud to provide uptime resiliency. A multi-node cluster ensures that NGINX Controller stays up even if one of the control-plane hosts becomes unavailable.
Set up data plane high availability (HA)

Configure a high-availability data plane for your apps in on-premises deployments using NGINX Controller, NGINX Plus, and keepalived. High-availability data planes help to ensure your apps operate continuously without service interruptions.

Support for High Availability (HA) mode is limited to two NGINX Plus instances.
Protect your apps with WAF

Now you can use the App Security add-on for the Application Delivery module to protect your apps with a web application firewall (WAF). The WAF protects your apps from various application-layer attacks such as cross-site scripting and SQL injection, among others. Organizations can start out-of-the-box with OWASP Top 10 protection with a default protection policy and minimal tuning required.

To get started, check out the Trial NGINX Controller with App Security guide.
Install NGINX Controller with an embedded config database

When installing NGINX Controller, you can choose to use an embedded, self-hosted config database by default, or you can provide your own external PostgreSQL database. If you use the embedded config database, NGINX Controller will install and configure the database for you, significantly streamlining the installation process.
Automated backups for the embedded config database

If you install NGINX Controller using the embedded, self-hosted config database, NGINX Controller will back up the database at hourly intervals for you automatically. For more information about these automated backups and how to restore from them, see the Backup & Restore Guide.
Improved user experience for configuring apps

We enhanced the user experience for configuring a basic app and components by reorganizing the settings pages to show standard configuration options first and advanced configuration options when needed.
Improved user experience for routing API resources

The user experience for routing API resources has been improved to make it easier to drag and drop multiple resources. Resources can now be checked individually and then dragged over together.
APIM adds support for response and request header modifications

Response and request headers can now be added to or deleted from published APIs.
Added support for NGINX Plus and Ubuntu
- Adds support for NGINX Plus R23
- Adds NGINX Controller Agent support for Ubuntu 20.04 LTS

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

All users require the built-in Guest role (6231)
Canceling an installation on a system running NGINX Controller will uninstall NGINX Controller (17676)
Type-1 directives ignored by UI for app components (17697)
Deleting a license using the NGINX Controller API does not remove the license from NGINX Controller (19090)

New Issues

The following issues are known to be present in this release. Look for updates to these issues in future NGINX Controller release notes.

Installation and Upgrade

Upgrading to NGINX Controller v3.12 fails on systems with less than an 8-core CPU (20509)

NGINX Controller requires an 8-core CPU @ 2.4 GHz or higher.

If you’ve installed NGINX Controller on a system that does not have an 8-core CPU @ 2.4 GHz or higher, you should not upgrade to v3.12.

Caution:

Installing or upgrading NGINX Controller on systems that do not meet the minimum hardware requirements may cause NGINX Controller to become unresponsive.

Refer to the NGINX Controller Tech Specs guide for hardware requirements and system recommendations.

Workaround:

Increase the number of CPUs to 8 or more @ 2.4 GHz or higher, then rerun the update script:
```
cd controller-installer
./update.sh
```
For help troubleshooting NGINX Controller upgrades, contact NGINX Support from the MyF5 Customer Portal.

NGINX Controller update script reports a permission denied error when loading CA certificate (21211)

When updating NGINX Controller, the update script may return an error similar to the following:

Error opening Certificate /etc/kubernetes/pki/ca.crt
139998249641872:error:0200100D:system library:fopen:Permission denied:bss_file.c:402:fopen('/etc/kubernetes/pki/ca.crt','r')
139998249641872:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load certificate
Error: Kubernetes control plane is not ready after updating cloud provider configuration

This error occurs if the Certificate Authority (CA) cert in /etc/kubernetes/pki/ca.crt does not have read permission set.

Workaround:

To allow read access for the CA file, run the following command:
```
sudo chmod 644 /etc/kubernetes/pki/ca.crt
```
Re-run the update script:
```
cd controller-installer
./update.sh
```

Analytics

The nginx.master.reloads metric increments and sums incorrectly (14533)

Previously the nginx.master.reloads metric remained 0 even when reloads were applied. Currently, this metric increases when reloads are not applied.

API Management

Object contains invalid IDP reference message (19943)

When upgrading from a version of NGINX Controller prior to and including v3.8, an error may be generated when updating a published API if it had an authentication policy.

Workaround:

Delete the published API and re-publish it.
To view Identity Providers and API Definitions, new users need at least guest role (20182)

New users will not see existing Identity Providers or API Definitions unless they are assigned one of the built-in roles (admin, user, guest). This is true even if the user is assigned a custom role with environment-level permissions.

Workaround:

When creating a new user, you must assign the guest role (or higher) for the user to view Identity Providers and API Definitions. Alternatively, if you’re using a custom role, you can use the API to grant explicit permissions for the /security/identity-providers and /services/api-definitions endpoints. See the Manage Roles and Role Groups topic details.

App Security

After enabling WAF, security violations aren’t reported right away (10558)

When an App Component is initially enabled with WAF, there may be a few seconds where Security Events (that is, WAF violation events) are not mapped to the correct App Component and App. The following warning message is logged in /var/log/nginx-controller/security-events-mgr.log: “Generating event without app-centric dimensions (i.e., app, component, environment, gateway, correlationId).”
Security events not mapped to App or App Component if combined length of resource IDs exceeds 445 characters (11112)

Security events might not be mapped to the correct App or App Component if the combined length of the resource IDs for Environment, App, App Component, and Gateway exceeds 445 characters.
NGINX service fails to start if SELinux enforcing mode is enabled after NGINX App Protect is deployed (16540)

Setting SElinux to enforcing mode on an NGINX App Protect Instance may cause the NGINX service to fail to start after the Instance is rebooted.

Workaround:

Enable SElinux enforcing mode before deploying NGINX App Protect.
The Instance hostname value is not available in the Security Events logs (17023)

The hostname value in the Security Events logs is incorrectly set to N/A.
Gateway configuration stuck at Configuring if referenced instances aren’t running App Protect processes (19441)

In the cases where a Gateway references multiple instances, the Gateway configuration status may be stuck at Configuring after an update request is made if the App Protect processes are not running on one or more of the instances that the Gateway references.

Workaround:

Ensure all the instances that the Gateway references have App Protect processes running, then make a request to update the Gateway configuration again.

Infrastructure

Controller Agent fails to download when deploying instance on AWS with RHEL 7 and CentOS 7 (19423)

When deploying an instance to AWS with RHEL 7 and CentOS 7, the installation fails, and the following error is logged to /var/log/agent_install.log: “Failed to download the install script for the agent.”

Workaround:

Ensure that the image referenced in the instance_template has wget installed.
NGINX Controller reports missing nginx.conf.bak file when reloading NGINX Plus configuration times out (20227)

In cases where reloading the backup NGINX Plus configuration file times out, NGINX Controller reports an error similar to the following:
```
Failed to deploy to instance (<instance name>). Could not restore: nginx.conf.bak rename /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf: no such file or directory.
```
This error is inaccurate and can be ignored. The nginx.conf.bak file cannot be found because the backup was successfully restored, and a subsequent attempt to restore from the file fails.

Licensing

APIM License limits may display incorrectly (19919)

After applying an NGINX Controller license, the license limits for API calls displayed in the web interface and the API are incorrect. You can safely ignore these incorrect license limits; you will have the correct entitlement on NGINX Controller.

Platform

Misleading error message when creating a node with a unique node name and an existing IP address (18838)

When creating a node with a unique name but with the same IP address as another node in a multi-node cluster, the following error is shown: “Error creating node: the node already exists. Use a different node name, then try again.”

Workaround:

When adding a node to a multi-node cluster, both the node’s name and IP address must be unique. If you encounter this error and you’ve used a unique name for the node, try using a different IP address.
Floating IP has to be removed to be updated (19011)

It’s not possible to swap out the floating IP with a different value and have the changes take effect immediately. To update the floating IP, you must first remove the existing value and then add a new one.

To change the floating IP, take the following steps:
1. To remove the existing floating IP using the NGINX Controller REST API, send a PATCH request to the /platform/global endpoint. Submit an empty string for the clusterFloatingIP value.
2. To remove the existing floating IP using the web interface:
  1. Open the NGINX Controller web interface and log in.
  2. Select the NGINX Controller menu icon, then select Platform.
  3. On the Platform menu, select Cluster.
  4. In the Cluster Configuration pane, select the edit icon (pencil).
  5. Select the Use Floating IP toggle to disable it.
  6. Select Save.
3. Restart NGINX Controller:
```
/opt/nginx-controller/helper.sh controller restart
```
Multi-node, resilient deployments and automated backups of embedded config DB are not supported on AWS (20052)

NGINX Controller does not support multi-node, resilient deployments on AWS at this time. Automated backups of the embedded config DB for multi-node, resilient clusters on AWS are also not supported.
Switching between roles in the Recent Roles and Groups menu shows incorrect permissions (20180)

If you select among the roles in the Recent Roles and Groups menu in the Platform web interface, the selected role’s displayed permissions are incorrect and are for the previously selected role. For example, if you select a role called “role-a,” and then select a role called “role-b”, the permissions shown for “role-b” are the permissions for “role-a.”

The permissions have not changed and will not change so long as you do not select Update.

Workaround:

To view the correct permissions for a custom role, select Roles Overview in the Roles menu, then select a role in the Roles list.

Documentation

API Reference does not specify that TCP/UDP App Components are experimental (20285)

In the NGINX Controller API Reference, the schema for App Components includes two objects related to TCP/UDP support that are missing the x-f5-experimental extension: ComponentTcpUdpDesiredState and ComponentTcpUdpCurrentState. TCP/UDP App Components are not supported in NGINX Controller v3.12.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R23
NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.10.0

October 27, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.10.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.10.0 includes the following updates:

Bug fixes and improvements.
Improves uninstalling NGINX Controller by deleting Kubernetes by default.
Optimizes health checks. Health checks are instantiated only once on an NGINX Plus instance for each upstream group of app servers.
Adds new request and response schema viewers to the Dev Portal, making it easier to understand each endpoint.
Adds support for API documentation on the Dev Portal when API definitions are published multiple times with different base paths.
Adds rate aggregation to the metrics API.
Adds an app-centric Analytics Overview Page.
Adds a new connected licensing option, which allows pricing based on usage.
Adds support for long hostnames – with a length of 255 octets, including separators – for Gateway URIs.

Resolved Issues

Vulnerability Fixes

The NGINX Controller Agent does not use absolute paths when calling system utilities (16119)
- Solution Article: K43530108 | CVE-2020-27730

Additional Fixes

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Selecting between recent API Definitions and Dev Portals doesn’t refresh the webpage (14602)
Once registered with NGINX Controller, a data path instance cannot be renamed (15835)
Create App Component settings for URI rewrites and header modification display URIs for non-referenced gateways (16103)
The NGINX Controller OpenAPI Spec fails OAS 3 validation (16366)
Increase the default PostgreSQL max database connections for production deployments (16386)
URI rewrites are not supported with API Key auth policy (16510)
Error tells users to check username and/or password when bind user credentials are incorrect (16602)
Long hostnames are not supported in Gateway URIs (16759)
The NGINX HTTP Errors graph per instance renders 4xx and 5xx error stats for upstreams in the same color (16884)
Add an Instance Using a Template procedure is missing a step (16920)

New Issues

Installation and Upgrade

Canceling an installation on a system running NGINX Controller will uninstall NGINX Controller (17676)

If you run the install.sh script on a system with NGINX Controller installed, and then press Ctrl+C to cancel the installation, the installer will roll back any changes and completely remove NGINX Controller.
On Red Hat Enterprise Linux, the /var/log/messages file fills up with errors related to db-rollup-job (17716)

On Red Hat Linux Enterprise (RHEL) systems running Docker CE and NGINX Controller, the /var/log/messages file is filled with errors related to db-rollup-job that look similar to the following:
```
docker_sandbox.go:384] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "db-rollup-job-1600982700-kpnzx_nginx-controller"
```
Workaround:

Docker CE is not officially supported on Red Hat Enterprise Linux (RHEL). To use Docker on RHEL, you must install Docker from the RHEL Extras repository.

Take the following steps to resolve this issue:

To resolve this issue:
1. Uninstall NGINX Controller:
```
/opt/nginx-controller/uninstall.sh
```
2. Remove Docker CE.
3. Install the supported Docker version from the RHEL Extras repository. See the Red Hat “Getting Started with Containers” guide for instructions.
4. Re-install NGINX Controller:
```
cd controller-installer
./install.sh
```

API Management

Specifying a URI to a JWK Set file is not supported on instances with multiple gateways (17124)

A JWT identity provider that’s configured to upload the JWT Set from a file referenced by a URI is not supported on NGINX Plus instances that have multiple gateways.

Workaround:

Ensure that only a single gateway resides on the NGINX+ instance.

Apps and Services

Type-1 directives ignored by UI for app components (17697)

There are many NGINX directives that are supported by the Components API that are not yet available within the web interface. To set any of these directives, you must make an explicit REST call to the Component endpoint. Because the web interface does not take this into account, any changes made to a Component via the web interface results in these settings being removed.

Workaround:

To take advantage of these new NGINX directives for Components, you must make any changes via the web interface first (or not use the web interface for Components). You can then use the explicit REST call for the Component to set the additional NGINX directives.

Infrastructure

System Properties dialog showing current and available Agent versions was removed (17998)

On NGINX Controller 3.10 and earlier, you could view the current Agent version and available Agent versions (for upgrades) by going to Infrastructure > Instance > Graphs, choosing a system in the list, and then selecting the gear icon to show the System Properties dialog. This dialog was removed in NGINX Controller 3.10.

Workaround:

When you upgrade NGINX Controller, you must upgrade the Controller Agent as well to keep the versions in sync, following the instructions here: Controller Agent Installation Guide.

To determine the Agent version that’s running on an instance, log into the instance and run the following command for your distribution type:
- On Ubuntu/Debian:
```
dpkg --list nginx-controller-agent
```
  The output looks similar to the following:
```
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii nginx-controller-agent 3.10.0-13964801~bionic amd64 NGINX Controller Agent
```
- On RHEL/CentOS:
```
sudo yum list installed nginx-controller-agent
```
  The output looks similar to the following:
```
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
* base: d36uatko69830t.cloudfront.net
* epel: d2lzkl7pfhq30w.cloudfront.net
* extras: d36uatko69830t.cloudfront.net
* updates: d36uatko69830t.cloudfront.net
Installed Packages
nginx-controller-agent.x86_64 3.10.0-13964792.el7 @nginx-controller
```
The leading part of the version string – that is, the part before the initial ‘-’ character (for example, 3.10.0) – should match your NGINX Controller vstring.

Platform

Users aren’t notified when an admin changes users’ passwords (18088)

When an administrator changes a user’s account password, the user is not notified about the password change. The user must contact the administrator to get the new password to log back into the system.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.9.0

September 24, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.9.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.9.0 includes the following updates:

Bug fixes and improvements.
Adds support for PostgreSQL 12.3.
Adds support for multiple gateway refs on a published API.
Adds the capability to publish API documentation simultaneously when you publish an API, keeping your docs and API in sync.
Adds the ability to redirect incoming HTTP requests, including the ability to associate redirect rules with specific component and/or gateway URIs.
Adds support for TLS tuning on the proxy side of the NGINX reverse proxy.
Adds support for acting on or ignoring specific response header fields from proxied servers.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

The Analyzer page reports 0 certificates when a cert is configured (14017)
NGINX Controller install script provides wrong commands for installing Docker CE on Red Hat Enterprise Linux 7 (15666)
Custom Dashboards must be re-created after upgrading from NGINX Controller 3.7 to 3.8 (15815)
Per-environment and guest user access is broken for upgrades from NGINX Controller 3.4 or earlier (15941)
NGINX Controller Agent troubleshooting guide incorrectly refers to stub_status module (16028)
After upgrading to NGINX Controller 3.8, some statistics aren’t updated on the Graphs page (16064)

New Issues

Analytics

The NGINX HTTP Errors graph per instance renders 4xx and 5xx error stats for upstreams in the same color (16884)

On the Infrastructure > Instances > Graphs page in the NGINX Controller web interface, the NGINX HTTP Errors graph per instance uses the same color for both 4xx and 5xx error statistics.

Workaround:

To see the 4XX errors, you can disable the 5XX errors by selecting http.status.5XX on the chart’s legend. Similarly, if you want to see the 5XX errors, disable the 4XX errors by selecting http.status.4XX on the chart’s legend.

API Management

URI rewrites are not supported with API Key auth policy (16510)

A published API cannot have URI rewrites and an API Key authentication policy specified in the same component.

Apps and Services

Long hostnames are not supported in Gateway URIs (16759)

Using long hostnames in the Gateway URI results in errors that cause the configuration to fail.

Infrastructure

Increase the default PostgreSQL max database connections for production deployments (16386)

When deploying NGINX Controller in production environments, the default PostgreSQL max_connections of 100 is too low to support more than 20 NGINX Plus instances.

Workaround:

Edit the PostgreSQL database configuration file to increase the max_connections value from the default of 100 to 200. See the PostgreSQL “Tuning your PostgreSQL Server” guide for instructions.
New instances stay in Configuring state if they exceed the license limit (16881)

When adding a data-path instance – for example, adding an NGINX Plus instance with the Controller Agent – that exceeds your license limit, the instance stays in a Configuring state and cannot be deleted. This unconfigured instance does not affect the ability to manage registered, licensed instances.

Workaround:

Contact NGINX Support via the MyF5 Customer Portal if you need to acquire more licenses. Once you have enough licenses, reinstall the Controller Agent on the instance that’s stuck in a Configuring state, using a different name for the instance than you used before. After the Controller Agent installs, the instance will register with NGINX Controller, where you’ll see the registered instance as well as the instance that’s stuck in a Configuring state. You can ignore the stuck instance.

Platform

Error tells users to check username and/or password when bind user credentials are incorrect (16602)

When Active Directory is used for external authentication, if the Bind user credentials or URI information are updated with invalid information, users are shown an error when logging on the NGINX Controller that their username or password is invalid. The error should say that the authentication provider cannot be reached and to try again later or contact the system administrator for assistance.

Documentation

The NGINX Controller OpenAPI Spec fails OAS 3 validation (16366)

The NGINX Controller OpenAPI specification that’s provided as a download from in the API Reference guide fails OAS 3 validation and cannot be imported into NGINX Controller.
Users endpoints aren’t shown in the NGINX Controller API reference guide (16705)

When viewing the NGINX Controller API reference guide, the Users endpoints might not display in the endpoints list. When scrolling to the bottom of the list, the list ends at Services.

Workaround:

Expand the Services endpoints. The endpoints list will refresh, and the Users endpoints are shown just below Services.
Add an Instance Using a Template procedure is missing a step (16920)

The Add an Instance Using a Template procedure is missing a step. The complete procedure is as follows:
1. Select the NGINX Controller menu icon, then select Infrastructure.
2. On the Infrastructure menu, select Instances.
3. On the Instances overview page, select Create.
4. Select Create a new instance using a template.
5. Add a name.
6. Select a Location in the list, or select Create New to create a Location.
7. Select an Instance Template in the list, or select Create New to create an Instance Template.
8. Select Submit.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.8.0

September 1, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.8.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.8.0 includes the following updates:

Bug fixes and improvements.
Adds stricter password requirements. Passwords for local users must be between 8–64 characters; commonly used passwords are rejected. All existing users must update their passwords to adhere to the updated password policy.
Adds SELinux support for data plane hosts.
Adds support for an internal PostgreSQL database for NGINX Controller, for demo and trial purposes only.
Adds support for basepath within a Published API.
Adds data forwarder for Datadog (beta).
Adds improved filtering for data forwarders.
Adds custom dashboards for Apps.
Adds full support for request URI rewrites for web apps, including:
- The ability to scope rewrite rules by associating rules with specific component and/or gateway URIs;
- The ability to specify rewrite flags.
Adds support for performance and interaction tuning (buffering, timeouts) between the NGINX+ data plane and the back-end (upstream) workloads.
Adds support for compressing responses using the “gzip” method. This often helps to reduce the size of transmitted data by half or even more.
Adds support for running the NGINX Plus data plane as a particular user.
Adds support for retrieving an NGINX Plus certificate and key via the REST API. You can download the certificate and key as a JSON or gzip file.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

The rewrite directive is not associated with a particular URI or location block in the generated configuration (10473)
SELinux settings can prevent statistics flow from agents (10342)
Code sample in the Dev Portal may have unintended URL encodings (13908)
Configuration changes are not deployed to all instances when one instance is unavailable (14227)
Note for PUT Update An Authentication Provider is incorrect (14576)
Cannot log in to the NGINX Controller web interface using Firefox v77 (14800)
Controller Agent fails to download when deploying an instance to AWS on Red Hat Enterprise Linux 7 (14886)
Upgrading Controller Agent returns an error that named instance already exists (14932)

New Issues

Installation and Upgrade

NGINX Controller install script provides wrong commands for installing Docker CE on Red Hat Enterprise Linux 7 (15666)

When installing NGINX Controller on Red Hat Enterprise Linux 7, the install script displays commands for installing Docker CE that are for are an older, unsupported version of Docker CE.

Workaround:

Before installing NGINX Controller on Red Hat Enterprise Linux 7, you should install Docker CE 18.09 (recommended version).
Custom Dashboards must be re-created after upgrading from NGINX Controller 3.7 to 3.8 (15815)

When upgrading from NGINX Controller 3.7 to 3.8, Custom Dashboards are not migrated. You must re-create any Custom Dashboards after completing the upgrade.

Analytics

Alerts set using NGINX Controller 3.6 or earlier need to be updated after upgrading to NGINX Controller 3.8 (14565)

Alerts that are set using NGINX Controller 3.6 and earlier are not checked after upgrading to NGINX Controller 3.8; the “last checked” dates for these alerts are from before the upgrade.

Workaround:

To resume checking the alerts, go to the Analytics > Alerts page in the NGINX Controller user interface, then mute and unmute the alerts.

Apps and Services

Create App Component settings for URI rewrites and header modification display URIs for non-referenced gateways (16103)

The Applicable URIs list for URI rewrites and header modifications on the Create App Component page shows both referenced and non-referenced gateway URIs to choose from. If the URI for a non-referenced gateway is selected, NGINX Controller returns an error like the following: “Applied URI: is not a part of the ingress URI.”

Workaround:

When selecting a gateway URI from the Applicable URIs list, select only URIs for gateways that the component references.

Infrastructure

The Analyzer page reports 0 certificates when a cert is configured (14017)

The Analyzer misevaluates SSL certificates associated with instances. The Analyzer page reports the number of certificates as 0 in the SSL block, even when certificates have been configured.

Workaround:

To see if your configuration is using a certificate:
1. Open the NGINX Controller web interface and log in.
2. From the NGINX Controller menu, select Services > Certs > Overview.
3. On the Certs Overview page, select a certificate from the list. This opens a side panel where you can view the certificate’s details, including whether the certificate is associated with an app component and/or gateway.
Once registered with NGINX Controller, a data path instance cannot be renamed (15835)

The name of your data-path instance – that is, the location where you install the NGINX Controller Agent – is fixed and cannot be changed once the instance is registered with NGINX Controller.

When upgrading the Controller Agent, the install script allows entering a different instance name; however, this name is meant for adding new instances only. Attempting to provide another instance name when upgrading the Controller Agent could cause metrics to be misreported.

Workaround:

When upgrading the Controller Agent you can omit the instance name, or if you include the name, you must use the existing instance name.
After upgrading to NGINX Controller 3.8, some statistics aren’t updated on the Graphs page (16064)

After upgrading from NGINX Controller 3.7 to 3.8, you may notice that some statistics on the Instances Graphs page – in the web interface, select Instances > Graphs – aren’t updated for some configurations.

Workaround:

To resolve this issue, restart the nginx and controller-agent services on the NGINX Plus instance:
- service nginx restart
- service controller-agent restart
After restarting these services, NGINX Controller should start reporting the statistics again.

Platform

Per-environment and guest user access is broken for upgrades from NGINX Controller 3.4 or earlier (15941)

Due to a change to the permission structure in NGINX Controller 3.4, upgrades from pre-3.4 versions to post-3.4 versions – including upgrades to NGINX Controller 3.8 – can break access for guest users and for users with “per-environment” roles.

Workaround:

There are two workarounds for this issue, depending on your upgrade history and risk tolerance.
- Option 1 (recommended): If you have not yet upgraded NGINX Controller from a pre-3.4 release, first upgrade to NGINX Controller 3.4, and then upgrade again to the latest release.
- Option 2: If you have already upgraded from a pre-3.4 release to a post-3.4 release without upgrading to 3.4 in the interim, you can only work around this issue by elevating the permissions of your guest users to regular users.
  
  Warning:
  This change involves a significant elevation of privilege and should be undertaken only with careful consideration. The regular user role has all the same admin role permissions except for the ability to promote new admins. If this change is acceptable for your installation – for any guest role users and any per-environment users with the guest role – these users should be edited by the default admin account to remove the guest role and replace it with the user role.

Documentation

NGINX Controller Agent troubleshooting guide incorrectly refers to stub_status module (16028)

In the NGINX Controller troubleshooting guide, which you can find at https://<Controller-FQDN>/docs/support/troubleshooting-controller, some of the information in the section covering metrics is incorrect.

See the following steps and corrections:
- Step 4. Make sure the Controller Agent is using the same user ID as the NGINX worker processes. The default user ID is nginx.
  
  Correction: The default user ID for the NGINX Controller Agent is root.
- Step 8. Make sure stub_status is configured, and the stub_status module is included in the NGINX build. To check, run the following command:
```
nginx -V
```
  The output should mention --with-http_stub_status_module.
  
  Correction: The NGINX Controller Agent gathers metrics via the NGINX Plus API, not via the stub_status module.
Workaround:

When troubleshooting issues with metrics, instead of checking that stub_status is enabled, you should verify that nginx api is enabled.
The applies-to version list at the bottom of some docs doesn’t include NGINX Controller 3.8 (16795)

At the bottom of the NGINX Controller docs, there’s a list of NGINX Controller versions that each doc applies to. For some docs installed with NGINX Controller 3.8, this list wasn’t updated. All of the docs installed with NGINX Controller 3.8 apply to that version.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.7.0

July 30, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.7.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.7.0 includes the following updates:

Bug fixes and improvements.
Bundles Kubernetes dependencies with the NGINX Controller installer, making installations more straightforward and reliable.
Disables port 80 for the NGINX Controller API gateway. Previously, NGINX Controller used port 80 to make initial redirects to HTTPS on port 443. Now, port 80 is disabled for the API gateway, and redirects aren’t supported. Port 443 is required to explicitly add https:// when pasting the FQDN in the address of the browser or when making API calls. Note: When you upgrade NGINX Controller, you also need to upgrade the NGINX Controller Agent software on each monitored NGINX Plus Instance.
Updates the Controller Agent to use golang instead of Python.
Adds Role Groups. Role Groups act as a collection of Roles that you can manage as a single resource. Role Groups can be mapped only to external auth provider groups; they cannot be assigned to local users.
Adds unique, individual salt to secure account passwords. Note: We recommend that users update their existing passwords to take full advantage of this enhancement.
Adds “Top X” functionality to limit the number of returned series in metrics queries to the top X data of probes. Multiple series that don’t fit into a limit are collapsed into a single series.
Adds a ‘dimension’ query parameter to the metrics API, which can be used to define the dimensions list to return for each metric series.
Adds a filtering option for refining the data that’s forwarded to Splunk using the data forwarder.
Introduces a new user interface for the Dev Portal. Create and manage beautiful, easy-to-use API reference documentation to support your published APIs.
Adds support for importing OpenAPI v2 specs (OAS) for your APIs.
Adds Total Latency metrics to the Environments, Apps, and App Components overview pages.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)
Certificates imported out of order using drag-and-drop cause an error when creating a cert bundle (10761)
Importing API specs fails when done in the web interface (13501)
Cannot log in to NGINX Controller when admin email address contains uppercase letters (13775)

New Issues

General

Cannot log in to the NGINX Controller web interface using Firefox v77 (14800)

When trying to log in to the NGINX Controller web interface using Firefox v77, the following error occurs: SyntaxError: invalid regexp group IdentityProvider.form.tsx:364:17 <anonymous> IdentityProvider.form.tsx:364 Webpack 28.

Workaround:

Upgrade to Firefox v78.

API Management

Code sample in the Dev Portal may have unintended URL encodings (13908)

In some cases, the example code snippet in the Dev Portal is URL encoded when the API has path parameters. For example, the code snippet might look similar to the following:
```
curl --request GET --url http://sportsapp.example.com/api/f1/drivers/%7BdriverId%7D
```
Workaround:

You must remove the URL encoding before running the example code snippet, like so:
```
curl --request GET --url http://sportsapp.example.com/api/f1/drivers/{driverId}
```
Known issues creating a Dev Portal using the NGINX Controller API (14385)
- When passing a partial or incomplete theme customization payload to Dev Portal CRUD operations, the Dev Portal uses the missing settings’ default values. To customize fully, include the complete payload.
- The Dev Portal supports only a curated list of fonts. You can see the list of accepted fonts while configuring a Dev Portal theme using the NGINX Controller web interface. If an invalid font value is submitted using the API, there’s no validation on the font value, and the default font is used instead.
Workaround:

Use the NGINX Controller web interface to construct a valid payload for creating and customizing the Dev Portal.
Error status of Published API component affects publishing to Dev Portal (14452)

The Dev Portal can be created and associated with a Published API at the same time. However, if the Published API is in an error state when the Dev Portal is created, an error is issued that incorrectly says that creating the Dev Portal failed. Then if the Dev Portal is resubmitted, the association fails with an “object exists” error. Furthermore, if the Published API is in an error state, publishing documentation to the Dev Portal for that API also fails.

Workaround:

Check the component status for the Published API before creating and associating a Dev Portal for it.
Selecting between recent API Definitions and Dev Portals doesn’t refresh the webpage (14602)

On the APIs page, selecting between items in the Recent API Definitions and Dev Portals quick list doesn’t refresh the page. Whichever item is displayed first is persistent.

Workaround:

To view specific API Definitions, select APIs > API Definitions. Then from the API Definitions list page, choose the API Definition that you want to view. Likewise, to view specific Dev Portals, select APIs > Dev Portals. Then from the Dev Portal list page, choose the Dev Portal that you want to view.

Infrastructure

Configuration changes are not deployed to all instances when one instance is unavailable (14227)

In multi-instance environments, when a configuration change cannot be deployed to one instance, such as when an instance is not available, the configuration is not deployed to any of the other instances.
Controller Agent fails to download when deploying an instance to AWS on Red Hat Enterprise Linux 7 (14886)

When deploying an instance to Amazon Web Services (AWS) on Red Hat Enterprise Linux 7, the installation fails, and the following error is logged to /var/log/agent_install.log: “Failed to download the install script for the agent.”

Workaround:
- Ensure that ports 443 and 8443 are open between NGINX Controller and the network where the instance is being deployed.
- Verify that NGINX Controller can be accessed from the instance using the FQDN that was provided when NGINX Controller was installed
- Ensure that the image referenced in the instance_template has a cURL version of 7.32 or newer.
Upgrading Controller Agent returns an error that named instance already exists (14932)

When Upgrading the Controller Agent, the install script warns that the named instance already exists and says to choose another name or remove the existing instance.

Workaround:

This error can be ignored for Controller Agent upgrades.

Documentation

Note for PUT Update An Authentication Provider is incorrect (14576)

In the NGINX Controller API reference guide, the description for the PUT /platform/auth/providers/{providerName} endpoint incorrectly says, “You must provide the full request body, including all configuration options, in a PUT request.”

As long as the required parameters are included in the PUT request, the update works.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.6.0

July 1, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.6.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.6.0 includes the following updates:

Bug fixes and improvements.
Adds external authentication using Active Directory (beta).

Note:
Active Directory users are given user-level access to NGINX Controller in this beta implementation.
Adds ability to create orchestrated Instances on AWS using Instance Templates.
Adds metrics forwarding for Splunk (beta).
Adds ability to secure published APIs by configuring authentication, conditional access, and rate limiting at the Component level.
Adds ability to configure DNS Service Discovery at the Component level.
Introduces a new user interface for Identity Providers.
Adds ./opt/nginx-controller/helper.sh repository-cred for getting the repo key and cert, needed for installing NGINX Plus.
Changes the default security option for Instance registration. Registration of NGINX Plus Instances is now performed over a secure connection by default. A new option – Allow insecure server connections to NGINX Controller using TLS – is available in the user interface. This option allows you to use self-signed certificates with the Controller Agent. For security purposes, we recommend that you secure the Controller Agent with signed certificates when possible.

End of Support: Debian 8 (i386, x86_64) is unsupported as of June 30, 2020. As such, NGINX Controller Agent support for Debian 8 will end with NGINX Controller v3.7. End of Support means that NGINX will not fix bugs related to Debian 8 in NGINX Controller 3.7 and later.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Command to install Controller Agent includes --no-check-certificate by default (4166)
- Solution Article: K31150658 | CVE-2020-5909
NATS services do not require authentication (6404)
- Solution Article: K59209532 | CVE-2020-5910
On Debian and Ubuntu, Kubernetes packages are downloaded over plaintext HTTP (11771)
- Solution Article: K84084843 | CVE-2020-5911

Additional Fixes

REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)
When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)
Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)
Component URI rewrites that are added and removed using the user interface cause a rewrite error (12527)
The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)
Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)
API resources with POST method must be defined using the NGINX Controller API (12652)

New Issues

Installation and Upgrade

Controller Agent warns that Instance name already exists during upgrade (13350)

When upgrading the Controller Agent, the install.sh script issues a warning that the Instance name already exists. This validation is a safeguard for new installations. You can safely disregard this warning for upgrades.
Cannot log in to NGINX Controller when admin email address contains uppercase letters (13775)

The installer for NGINX Controller allows users to create admin accounts with email addresses that contain uppercase letters. However, NGINX Controller expects emails to be only lowercase and prevents admin accounts with differently formatted email addresses from logging in.

Workaround:

You can reinstall NGINX Controller and enter a lowercase email address for the admin account. If that doesn’t work, contact NGINX Support for assistance.

API Management

Published APIs do not go into an error state when the Instance they are using is deleted (10616)

If you delete the last Instance serving a published API, and then edit the published API, the published API gets stuck in the Configuring state. Attempting to delete the published API causes the published API to get stuck in a Deleting state.
Rate-limit policies are lost after upgrading to NGINX Controller v3.6 (12962)

Existing rate-limiting policies applied to published APIs are lost when upgrading to NGINX Controller 3.6. The policies must be reconfigured at the Component level following the upgrade.
JWK files need to be resubmitted after upgrading from NGINX Controller v3.4 to v3.6 (13289)

Existing JWK files applied to Identity Providers are lost when upgrading from v3.4 to v3.6 and must be resubmitted following the upgrade.
Importing API specs fails when done in the web interface (13501)

Importing API specs from within the NGINX Controller user interface fails because of how the interface parses the text.

Workaround:

Import API specs using the NGINX Controller REST API.

Infrastructure

Restarting the Controller Agent service leaves behind PID file with errors logged to agent.log (13390)

When the Controller Agent service is restarted, the controller-agent.pid file is left behind, and errors about the PID file are logged in agent.log.

Workaround:

Manually remove /var/run/controller-agent/controller-agent.pid, then restart the Controller Agent service again. The Controller Agent will work normally after it restarts.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.5.0

June 9, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.5.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.5.0 includes the following updates:

Bug fixes and improvements.
Adds support for NGINX Controller on Red Hat Enterprise Linux 7.
Deprecates support for NGINX Controller Agent on CentOS 6.
NGINX Controller requires the yum-plugin-versionlock package on Red Hat Enterprise Linux and CentOS.
Adds Gateway dimension to the metrics API.
Adds proxy parameters for Gateways and Components to manage retry attempts on error conditions.
Adds socket parameters for Gateways and Components to manage incoming TCP connections for the data plane.
Introduces a new user interface and declarative API for API Management.
Adds support for importing an OpenAPI specification when creating an API Definition.
Adds versioning for your API Definitions.
The Global Settings endpoint supports configuring API Gateway certificates and database settings.
Group your NGINX Instances by their physical locations using the Locations setting.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

The NGINX Controller web interface doesn’t have sufficient protection against a Cross-Site Request Forgery (CSRF) attack. (4882, 9721)
- Solution Article: K31044532 | CVE-2020-5900
The account recovery code for NGINX Controller that’s sent in email is not secure (11092)
- Solution Article: K25434422 | CVE-2020-5899
Undisclosed API endpoints may allow for a reflected Cross-Site Scripting (XSS) attack (12021)
- Solution Article: K43520321 | CVE-2020-5901

Additional Fixes

Cannot delete an instance that doesn’t have NGINX Plus installed (7704)
NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)
Cannot change SMTP password using the platform/global endpoint (11049)
NGINX Controller v3 does not support Red Hat Enterprise Linux 7 (11282)
Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)
Security section appears in the documentation but has no content (11294)

New Issues

Installation and Upgrade

Package error when installing nginx-plus-module-njs on CentOS 7 (12638)

When installing nginx-plus-module-njs on CentOS 7, the installer returns a package error.

Workaround:

Uninstall the existing nginx-plus-module-metrics module:
```
sudo yum remove -y nginx-plus-module-metrics
```
Install the latest version of nginx-plus-module-njs:
```
sudo yum install -y nginx-plus-module-njs
```
Install the nginx-plus-module-metrics module, specifying the version as 1.19:
```
sudo yum install -y nginx-plus-module-metrics-1.19*
```

Verification:

To confirm that the correct versions are installed, run the following command:

yum list installed | egrep 'nginx|metrics'

The expected output looks similar the following:

avrd-libs.x86_64 0.15.5_1850598.release_3_5-1 @nginx-controller-metrics
nginx-controller-agent.x86_64 3.5.0-10484161.el7 @nginx-controller
nginx-plus.x86_64 22-1.el7.ngx @nginx-plus
nginx-plus-module-metrics.x86_64 1.19.0-1850598.el7.ngx @nginx-controller-metrics
nginx-plus-module-njs.x86_64 22+0.4.1-1.el7.ngx @nginx-plus

API Management

The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)

The URIs for a Published API use the PREFIX match method, regardless of the resource match method that was specified in the API Definition.
API resources with POST method must be defined using the NGINX Controller API (12652)

When defining an API Definition version with a resource that uses the POST method and specifies an example request, the generated API spec in the web interface is invalid.

Workaround:

Use the NGINX Controller API to define API Definition version resources that use the POST method and contain an example request in the API documentation.

Apps and Services

Component URI rewrites that are added and removed using the web interface cause a rewrite error (12527)

If a URI rewrite for a Component is added and then deleted using the NGINX Controller web interface, the API spec that’s sent to NGINX Controller includes an empty programmability section. This results in an Error status for the Component, and an error similar to the following is returned: "Error: Arguments of rewrite : incomingPattern or rewritePattern is missing."

Workaround:
- Option 1 (Recommended): Send a PUT request to the Components endpoint that excludes the programmability section.
- Option 2: Configure a placeholder rewrite that writes from a pattern to a pattern. Using a pattern that doesn’t match any request is best.

Infrastructure

Controller Agent installation doesn’t warn about duplicate instance names until an Instance has been registered with NGINX Controller (12370)

The Controller Agent installation script typically fails with a warning if the provided instance name already exists for the specified location. However, a race condition occurs if the installation script is run on two instances at the same time with the same name for the same location. When this happens, the failed installation doesn’t issue a warning, but only one instance is added to NGINX Controller.

Workaround:

No workaround is necessary. The behavior is the same as if the installation script displayed the warning.
Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)

When the nginx-controller-agent package is removed on an NGINX Plus instance using apt-get-purge, the /etc/controller-agent directory and its contents are deleted. This directory is where NGINX Controller places certificates when NGINX Controller is used to configure and manage NGINX Plus instances. When this directory is deleted, the NGINX Plus instance becomes unresponsive and the NGINX service fails to start.

Platform

Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)

When a database certificate is added that uses an IP address instead of an FQDN, or if the certificate Common Name (CN) is invalid, the NGINX Controller database services reject the invalid cert and refuse database connections. In this case, the system becomes unresponsive until a valid cert is provided.

Workaround:

Use the hostname instead of an IP address, or use a certificate where the IP address is mentioned in Subject Alternative Name (SAN) as DNS type.
NGINX Controller Agent and AVRD do not work on 32-bit distributions (12605)

The Analytics, Visibility, and Reporting daemon (AVRD) is compiled as a 64-bit app and does not run on 32-bit distributions. Configuration Management using the Controller Agent is possible on 32-bit distributions; however, metrics are not reported.

Workaround:

Install the NGINX Controller Agent and AVRD on a 64-bit distribution.

Important:
NGINX Controller support for 32-bit distributions will be deprecated in a future release.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R22
NGINX Plus R21
NGINX Plus R20
NGINX Plus R19

NGINX Controller v3.4.0

May 6, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.4.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.4.0 includes the following updates:

Bug fixes and improvements.
Improved Controller Agent error messages that make it easier to troubleshoot connectivity issues during installation.
Write permissions for the platform/global endpoint are limited only to the Admin role now, to safeguard against changes to critical server and database configurations.
Audit events are exposed using the /events endpoint.
Write permissions for the platform/global endpoint are limited only to the Admin role now, to safeguard against changes to critical server and database configurations.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Malformed messages lead to segmentation fault of the Analytics, Visibility, and Reporting daemon (AVRD) (9068)
- Solution Article: K95120415 | CVE-2020-5895
The Analytics, Visibility, and Reporting daemon (AVRD) is world-readable and writeable (9067)
- Solution Article: K95120415 | CVE-2020-5895
The NGINX Controller webserver does not invalidate the the server-side session token when users log out (8576)
- Solution Article: K13028514 | CVE-2020-5894

Additional Fixes

API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

New Issues

Installation and Upgrade

Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)

When the conditions noted below are met, the Identity Provider API keys are not migrated and must be re-entered.

Conditions:

This issue applies when the following conditions are true:
- You are running NGINX Controller v3.0.
- You are running NGINX Controller v3.2, which you previously upgraded from NGINX Controller v3.0.
- You are running NGINX Controller v3.3, which you previously upgraded from NGINX Controller v3.0.
Workaround:

Step 1: Copy and save your API Identity Profile keys

Before upgrading, you must complete the following steps to copy and save any API Identity Profile keys that you have.
1. Open the NGINX Controller web interface and log in.
2. Select the NGINX Controller menu icon, then select Services.
3. On the Services menu, select APIs.
4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
7. Copy the Identity Provider API key value and save it locally. You’ll need to re-enter this value after you upgrade.
You can now complete the upgrade to NGINX Controller v3.4.

Step 2: Restore your API Identity Profile keys

After you’ve upgraded to NGINX Controller v3.4, you can restore the API Identity profile keys that you saved.
1. Open the NGINX Controller web interface and log in.
2. Select the NGINX Controller menu icon, then select Services.
3. On the Services menu, select APIs.
4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
7. Paste the copied Identity Provider API key that you saved in the previous procedure into the key field. Then, click Save.
NGINX Controller v3 does not support Red Hat Enterprise Linux 7 (11282)

NGINX Controller v3 does not support Red Hat Enterprise Linux (RHEL) v7. The NGINX Controller installation fails during the Docker version check. Manually updating the Docker version on Red Hat Enterprise Linux 7 does not solve the problem.

Apps and Services

Certificates imported out of order using drag-and-drop cause an error when creating a cert bundle (10761)

When certificates are imported using the drag-and-drop feature in the NGINX Controller web interface, the system expects the first certificate that’s added to be the public certificate. Additional certificates that are added are treated as intermediate CA certs.

If the certificates aren’t added in order, with the public certificate added first, the resulting API request has transposed values for the publicCert and caCert parameters.

Workaround:

When adding certificates using drag-and-drop, always drag the public cert file in first, followed by any additional CA certificates. The private key can be dragged in either before or after the certificates. Alternatively, you can choose to copy and paste the PEM text into the corresponding fields.

Platform

Cannot change SMTP password using the platform/global endpoint (11049)

Sending the SMTP password in a PATCH request to the platform/global endpoint has no effect.

Workaround:

To change the SMTP password, you can use the help.sh script that’s located in /opt/nginx-controller/ on the NGINX Controller host. For instructions, see the documentation that’s installed with NGINX Controller: https://<Controller-FQDN>/docs/platform/using-helper-script/#update-smtp-settings.

Documentation

Security section appears in the documentation but has no content (11294)

In NGINX Controller v3.4, a “Security” panel appears in the “Services” section of the onboard documentation. If you click on the Security panel, you will not see any content listed because there is no security documentation for this release. Users should disregard the “Security” panel in the documentation for this release.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R19
NGINX Plus R20
NGINX Plus R21

NGINX Controller v3.3.0

April 09, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.3.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.3.0 includes the following updates:

Bug fixes and improvements.
Adds support for certificate chains for TLS connections.
Adds an API catalog for the Metrics API.
Adds new dimensions for filtering metrics on the Application Summary page.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

The NGINX Controller Agent installer script ‘install.sh’ uses HTTP instead of HTTPS to check and install packages (8587)
- Solution Article: K00958787 | CVE-2020-5867
The helper.sh script, which is used optionally in NGINX Controller to change settings, uses sensitive items as command-line arguments (7436)
- Solution Article: K11922628 | CVE-2020-5866
NGINX Controller uses insecure connection for communications with PostgreSQL database by default (6509)
- Solution Article: K21009022 | CVE-2020-5865

Additional Fixes

Creation of eventually consistent objects returns 201 instead of 202 (7052)
Can’t delete a configured gateway when there is a gateway in an error state (7058)
Dependencies Missing from Open Source Dependencies Report (7404)
Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)
Intermediate certificates not pushed to NGINX Plus instance (7500)
Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)
Agent warns about failed connection to receiver after update (9152)
Certificate expiration shows expired warning and expiration date that doesn’t match the final certificate (9167)
Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)
Login endpoint has inaccurate session information in the API Reference Guide (9482)
A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

New Issues

Apps and Services

NGINX Controller web interface shows “Error fetching apps” after a restart (10013)

In some cases, after NGINX Controller restarts, you might see an error in the web interface that says, “Error fetching apps: Network Error.”

Workaround:

Log out of the web interface and then log back in.
Gateways configured with duplicate ingress URIs error out with condition of “config” (10425)

When upserting a Gateway, if the ingress URI (desiredState.ingress.uris) is a duplicate of another Gateway’s ingress URI, the upsert errors out as expected; however, the condition type (state.conditions.type) that’s reported is simply config, instead of an explanation for the condition that’s occurred.
```
"state": {
    "conditions": [
        {
            "type": "config"
        }
    ],
    "selfConfigState": {
        "configured": 0,
        "configuring": 0,
        "deleting": 0,
        "error": 1,
        "isConfigured": false,
        "isConfiguring": false,
        "isDeleting": false,
        "isError": true,
        "total": 1
    }
}
```
Workaround:

To resolve the error, remove the Gateway with the duplicate ingress URI that’s in the error state. You can do this by issuing a PUT request and applying a new config with a unique ingress URI, or you can issue a DELETE request to remove the Gateway with the duplicate ingress URI.
Component fails to configure when its Gateway URI collides with another Gateway’s URI (10430)

In situations where a Component for one Gateway uses the FQDN of a different Gateway on the same host, NGINX Controller doesn’t push the config to NGINX Plus.

Scenario
1. Create a Gateway for foo.com with the hostname http://foo.com, and deploy to NGINX Plus ‘Instance 2’.
2. Create a Component for foo.com with the URI http://foo.com/, and associate the Component with the Gateway foo.com and assign a workload.
3. Create a Gateway for bar.com with the hostname http://bar.com, and deploy to NGINX Plus ‘Instance 2’.
4. Create a Component for bar.com with the URI http://foo.com/baz, and associate the Component with the Gateway bar.com and assign a workload.
In this scenario, the configuration should expose the location for /baz. However, although the Component is reported as configured, nothing happens.
App Services API Known Issues (10473)

The App Services API has known issues for certain attributes and objects, in particular:
- Socket parameters defined in the API should not be used at all. This includes the receivBufferSize and sendBufferSize parameters that are common to Gateway ingress and Component ingress.
- Session persistence works only for SessionPersistenceCookie. Using SessionPersistenceRoute or SessionPersistenceCookieLearn results in configuration errors.
- When configuring workload monitoring, and when using the API to check headers in the response, it’s necessary to check for an exact match for the header. Matching with a regular expression or checking only for the existence of a header does not work.
- A Load Balancing method of HASH won’t work if it contains a variable such as $request_URI for the userKey.
The existing URI Rewrite capability in the App Services API has the following limitations:
- The Rewrite directive is not associated with a particular URI or location block in the generated configuration.
- The break flag is not supported.

Analytics

SELinux settings can prevent statistics flow from agents (10342)

Depending on the SELinux settings, an NGINX instance might not have permission to submit statistics to your NGINX Controller.

If you have a properly configured environment, application, and component for which no metrics are being passed to NGINX Controller after more than 30 seconds of traffic have transpired, try the suggested workaround steps.

See the NGINX blog post Using NGINX and NGINX Plus with SELinux for additional general guidance.

Workaround:
1. On the instance(s) associated with your Gateway, run the following command as root:
```
# grep nginx /var/log/audit/audit.log | audit2allow -M nginx`
```
2. Then, run this command:
```
# semodule -i nginx.pp
```

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R19
NGINX Plus R20
NGINX Plus R21

NGINX Controller v3.2.0

March 17, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.2.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.2.0 includes the following updates:

Bug fixes and improvements.
API definition URIs support substitutions. When creating or editing a URI within an API Definition, you can enter substitution patterns by using the {} notation.
Adds support for tagging instances.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

API allows for unauthenticated account creation (9428)
- Solution Article: K14631834 | CVE-2020-5863
NGINX Controller Agent uses insecure connection for communications between NGINX Controller and NGINX Plus instances by default (5899)
- Solution Article: K27205552 | CVE-2020-5864

Additional Fixes

Stack trace error occurs when accessing the Login credentials schema in the API reference docs. (6409)
JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)
Links to metrics descriptions from user interface point to the metrics reference landing page (7255)
Usage of API key authentication policies requires the njs module on data-plane instances (7298)
Omitting URI when defining a component may result in unexpected behavior (7308)
Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

New Issues

Apps and Services

Certificate expiration shows expired warning and expiration date that doesn’t match the end-entity certificate (9167)

A certificate might appear as expired on the certificate detail page, although the Valid To date is in the future. This happens because the summary status and expiration date are calculated based on the entire certificate chain from which the certificate is derived. So, if you have an intermediate CA cert in your chain, the expiration summary status and summary expiration date may reflect the status/date of any intermediate certificate. If you see such a status or expiration date, you should check the status of your entire certificate chain, including that of any intermediate CA.
A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

A high volume of Create, Update, and Delete (CUD) operations – for example, more than 90 – performed on Gateways and Components could create a condition in which NGINX Controller becomes unresponsive.

NGINX Controller limits queued operations at 100 and then returns an HTTP 429 error: “Too many requests; wait, and try again.” If, subsequently, additional CUD operations are submitted at a rapid rate (more than 1 per second), the queue may be overrun, creating a condition where no further changes can be processed without involvement from NGINX Support.

Workaround:

To avoid this situation, limit the amount of API Create, Update, and Delete (CUD) operations sent to the Component and Gateway endpoints to fewer than 100 open actions. The queue should clear 1 action per 7 seconds, on average.

Infrastructure

Agent warns about failed connection to receiver after update (9152)

After updating NGINX Controller and the Controller Agent to v3.2.0, the system warns that the Agent “Failed to connect to the receiver.” However, if you look in the agent.log file, the Agent is logging communication with the receiver, as expected.

This warning occurs when a TLS certificate signed with a custom CA is used for the NGINX Controller. Before v3.2.0, the Controller Agent did not verify server certificates.

To determine the status of the Agent, perform the following on the Agent host:
- Check the log file /var/log/nginx-controller/agent.log for errors.
- Check the status of the Controller Agent service:
  
  systemctl status controller-agent
  
  The output should include the status running.
Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)

If a TLS certificate for NGINX Controller is signed with a custom CA, the Agent fails to verify the cert and returns an error similar to the following:
```
2020-03-09 20:38:28,803 [5089] MainThread failed POST "https://<fqdn>:8443/1.4/<uuid>/agent/",
exception: "HTTPSConnectionPool(host='<fqdn>', port=8443): Max retries exceeded with url: /1.4/<uuid>/agent/
(Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))"
```
Workaround:
1. Place the CA on the Agent host.
2. On the Agent host, edit the /etc/controller-agent/agent.conf file. In the cloud directive section, make the following changes:
  - Add or set the requests_ca_bundle option to point to the CA file.
  - Set verify_ssl_cert to True. (Note: If this setting is set to False, then server certificate validation is disabled, which we don’t recommend.)
```
[cloud]
...
verify_ssl_cert = True
requests_ca_bundle = <path-to-ca-bundle-file>
```
3. Save the changes to the agent.conf file.
4. Restart the agent:
  
  systemctl restart controller-agent
5. Verify that the Agent can connect to the NGINX Controller host:
  
  systemctl status controller-agent
  
  The output should include the status running. You can also check the log file /var/log/nginx-controller/agent.log for errors.

Documentation

Login endpoint has inaccurate session information in the API Reference Guide (9482)

The API reference documentation for the Login endpoint describes new session behavior that hasn’t been implemented yet. At this time, the default maximum session lifetime is 8 hours. Once the session lifetime limit is exceeded, you must log in to obtain a new session token.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R19
NGINX Plus R20

NGINX Controller v3.1.0

February 26, 2020

These release notes provide general information and describe known issues for NGINX Controller v3.1.0, in the following categories:

Updates
Resolved Issues
New Issues
Supported Versions

Updates

NGINX Controller 3.1.0 includes the following updates:

Bug fixes and improvements.
The Components resource adds experimental strategyRef and waf placeholder objects.

The strategyRef and waf objects for the Components resource are experimental placeholder objects. Including the strategyRef and/or waf blocks in the component specification should not enable additional security-related application traffic services, even if there are no errors. If you see the error “Security module not found on the NGINX instance,” you should remove these blocks.

Note:
The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension.
This release adds new app-centric metrics and a new summary page in the UI for viewing these metrics. You can use these app metrics to monitor and evaluate the condition of your apps.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Cannot add tags to users with the NGINX Controller web interface (7165)
Gateway accepts a relative path instead of a URI (7212)
A component with no URIs will result in an errored deployment (7311)
NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)
Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)
Referencing a non-existing identity provider for a component does not generate an error (7397)
GUI incorrectly shows default error log state as enabled (7447)
Component-based access logging format specification is non-functional (7646)
Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)
Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)
API definitions base path is not imported from OAS spec file (7671)
The API Reference documentation omits the “Get a Location” endpoint (7747)

New Issues

Installation and Upgrade

Could not initialize a Kubernetes cluster when installing NGINX Controller on CentOS 7 (5863)

There is an issue in CentOS 7 that creates a mismatch between the optimum Docker settings and optimum Kubernetes settings. This mismatch prevents Kubernetes from initializing during the installation of NGINX Controller on CentOS 7.

The NGINX Controller installation fails after Docker and Kubernetes are installed, with the following error: “execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster.”

Workaround:
- Solution one:
  - Run sudo yum update. This updates the components to the latest version and patches the issue.
- Solution two:
  1. Run controller-installer/install.sh and follow the instructions. Allow the installation to fail. This establishes Docker and Kubernetes.
  2. Stop Docker: sudo systemctl stop docker
  3. Edit /etc/docker/daemon.json and make the the following change: native.cgroupdriver=systemd to native.cgroupdriver=cgroupfs
  4. Start Docker: sudo systemctl start docker
  5. Reset the Kubernetes configuration: sudo kubeadm reset -f
  6. Run controller-installer/install.sh and follow the instructions. Rerunning the installation does not update the version of any of the packages on the machine.

Apps and Services

When committing a gateway with REGEX matchMethod, whose URI includes an explicit port and a path, the error message incorrectly states that the port is invalid (8452)

In the context of configuring a gateway, only the protocol, domain, and optional port have any meaning. If the URI specified when configuring a REGEX matchMethod gateway includes a path, including a final / after the port, you will receive an error stating: Port value is invalid for hostname .*\\.example\\.com, where the hostname string corresponds to the data you provided for the domain.

Workaround:
- Specify your REGEX URI for a gateway as only the protocol, domain, and port with no trailing path.
REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)

The logic for interpreting REGEX and REGEX_CASE_SENSITIVE match methods for REGEX-based Component URIs is reversed.

REGEX-based Component URIs are converted into a case-insensitive location block in nginx.conf when the URI is specified as a relative URI (leads with a path, not a domain) and when the match method is specified as REGEX_CASE_SENSITIVE. Conversely, when the match method is REGEX, the REGEX-based Component URI is interpreted as case sensitive.

If you need your REGEX-based Component URI to be interpreted as case sensitive, then you can specify the match method as REGEX for now. However, keep in mind that when this logic mismatch is fixed in a later release of NGINX Controller, you’ll need to revert the match method accordingly.
When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)

In some cases, when communication between NGINX Controller and the Controller Agent is interrupted, an error similar to the following is returned: “Error: Failed to deploy to instance, Could not find the targeted NGINX running on the system None”. This error neglects to indicate the name of the system or systems for which the communication has failed.

Workaround:
- Communication difficulties can be persistent or temporary, depending on the nature of the failure. It is possible to see some more persistent communication failures by looking at the Infrastructure > Graphics pane in NGINX Controller. If the hostname and nginx-plus instance are down, these items will appear with a red background.
- If all the systems and nginx-plus instances have with a white background, the communication problem could be temporary or intermittent, and won’t necessarily show in the interface.
  
  Things to check in order are:
  - Ensure the agent host/container is up
  - Ensure the controller-agent process is running
  - Ensure the nginx-plus process is running
  - Verify network communication between NGINX Controller and the Controller Agent on port 443.

Analytics

Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)

When upgrading from NGINX Controller 3.0.0 to 3.1.0, the app-centric metrics are not sent from the NGINX instances to NGINX Controller until a new configuration push is performed.

Workaround:
- Perform a Put to an object on each NGINX instance to update the config.

Documentation

API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

The API Management documentation states that the API Management module requires a dedicated NGINX Plus instance. This is no longer the case. As of v3.0, the API Management module no longer needs to run on a dedicated instance.
Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

The following API actions are considered “experimental” features; use is not supported and functionality may change.
- Create an Environment
- Create an App
- Create or update an App Component with the following properties defined in desiredState:
  - ingress.tcpKeepAlive
  - backend.workloadGroups.uris.failTimeout
  - backend.workloadGroups.uris.resolve
  - backend.workloadGroups.uris.route
  - backend.workloadGroups.uris.service
  - backend.workloadGroups.uris.slowStart
  - backend.workloadGroups.serviceDiscovery
  - backend.workloadGroups.serviceDiscovery.dnsResolver
  - backend.workloadGroups.serviceDiscovery.dnsResolver.uris
  - backend.preserveHostHeader
  - backend.queue
  - backend.httpVersion
  - backend.ntlmAuthentication
  - backend.persistentState
  - programmability.httpHttpsRedirect
  - programmability.uriRedirects
  - programmability.cookieModifications
  - programmability.requestHeaderModifications
  - programmability.responseHeaderModifications
  - security
  - security.waf
  - security.conditionalAuth
  - security.interceptWorkloadErrors
- Create or update a Gateway with the following properties defined in desiredState:
  - ingress.headers
  - ingress.http2
  - ingress.spdy
  - ingress.proxyProtocol
  - ingress.setFib
  - ingress.fastOpen
  - ingress.backlog
  - ingress.acceptFilter
  - ingress.deferred
  - ingress.isIpv6Only
  - ingress.reusePort
  - ingress.notFoundStatusCode
Stack trace errors occur in the API reference docs (8481)

In NGINX Controller, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/.

For a number of elements, clicking on an object to view the nested content will result in a ReDoc stack trace error.

Workaround:

Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.
Dependencies Missing from Open Source Dependencies Report (7404)

In NGINX Controller v3.1, the following open source dependencies were omitted from the open source acknowledgements PDF:

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

NGINX Plus R19
NGINX Plus R20