NGINX Documentation

NGINX Controller Releases

For additional release documentation, see the following publications:


NGINX Controller Version 3.9.0

September 24, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.9.0, in the following categories:

Updates

NGINX Controller 3.9.0 includes the following updates:

  • Bug fixes and improvements.
  • Adds support for PostgreSQL 12.3.
  • Adds support for multiple gateway refs on a published API.
  • Adds the capability to publish API documentation simultaneously when you publish an API, keeping your docs and API in sync.
  • Adds the ability to redirect incoming HTTP requests, including the ability to associate redirect rules with specific component and/or gateway URIs.
  • Adds support for TLS tuning on the proxy side of the NGINX reverse proxy.
  • Adds support for acting on or ignoring specific response header fields from proxied servers.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

  • The Analyzer page reports 0 certificates when a cert is configured (14017)
  • NGINX Controller install script provides wrong commands for installing Docker CE on RHEL 7 (15666)
  • Custom Dashboards must be re-created after upgrading from NGINX Controller 3.7 to 3.8 (15815)
  • Per-environment and guest user access is broken for upgrades from NGINX Controller 3.4 or earlier (15941)
  • NGINX Controller Agent troubleshooting guide incorrectly refers to stub_status module (16028)
  • After upgrading to NGINX Controller 3.8, some statistics aren’t updated on the Graphs page (16064)

Known Issues

Analytics

  • The NGINX HTTP Errors graph per instance renders 4xx and 5xx error stats for upstreams in the same color (16884)

    On the Infrastructure > Instances > Graphs page in the NGINX Controller web interface, the NGINX HTTP Errors graph per instance uses the same color for both 4xx and 5xx error statistics.

    Workaround:

    To see the 4XX errors, you can disable the 5XX errors by selecting http.status.5XX on the chart’s legend. Similarly, if you want to see the 5XX errors, disable the 4XX errors by selecting http.status.4XX on the chart’s legend.

Infrastructure

  • New instances stay in Configuring state if they exceed the license limit (16881)

    When adding a data-path instance – for example, adding an NGINX Plus instance with the Controller Agent – that exceeds your license limit, the instance stays in a Configuring state and cannot be deleted. This unconfigured instance does not affect the ability to manage registered, licensed instances.

    Workaround:

    Contact NGINX Support via the MyF5 Customer Portal if you need to acquire more licenses. Once you have enough licenses, reinstall the Controller Agent on the instance that’s stuck in a Configuring state, using a different name for the instance than you used before. After the Controller Agent installs, the instance will register with NGINX Controller, where you’ll see the registered instance as well as the instance that’s stuck in a Configuring state. You can ignore the stuck instance.

Documentation

  • The NGINX Controller OpenAPI Spec fails OAS 3 validation (16366)

    The NGINX Controller OpenAPI specification that’s provided as a download from in the API Reference guide fails OAS 3 validation and cannot be imported into NGINX Controller.

  • Users endpoints aren’t shown in the NGINX Controller API reference guide (16705)

    When viewing the NGINX Controller API reference guide, the Users endpoints might not display in the endpoints list. When scrolling to the bottom of the list, the list ends at Services.

    Workaround:

    Expand the Services endpoints. The endpoints list will refresh, and the Users endpoints are shown just below Services.

  • Add an Instance Using a Template procedure is missing a step (16920)

    The Add an Instance Using a Template procedure is missing a step. The complete procedure is as follows:

    1. Select the NGINX Controller menu icon, then select Infrastructure.
    2. On the Infrastructure menu, select Instances.
    3. On the Instances overview page, select Create.
    4. Select Create a new instance using a template.
    5. Add a name.
    6. Select a Location in the list, or select Create New to create a Location.
    7. Select an Instance Template in the list, or select Create New to create an Instance Template.
    8. Select Submit.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21
  • NGINX Plus R22

NGINX Controller Version 3.8.0

September 1, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.8.0, in the following categories:

Updates

NGINX Controller 3.8.0 includes the following updates:

  • Bug fixes and improvements.
  • Adds stricter password requirements. Passwords for local users must be between 8–64 characters; commonly used passwords are rejected. All existing users must update their passwords to adhere to the updated password policy.
  • Adds SELinux support for data plane hosts.
  • Adds support for an internal PostgreSQL database for NGINX Controller, for demo and trial purposes only.
  • Adds support for basepath within a Published API.
  • Adds data forwarder for Datadog (beta).
  • Adds improved filtering for data forwarders.
  • Adds custom dashboards for Apps.
  • Adds full support for request URI rewrites for web apps, including:
    • The ability to scope rewrite rules by associating rules with specific component and/or gateway URIs;
    • The ability to specify rewrite flags.
  • Adds support for performance and interaction tuning (buffering, timeouts) between the NGINX+ data plane and the back-end (upstream) workloads.
  • Adds support for compressing responses using the “gzip” method. This often helps to reduce the size of transmitted data by half or even more.
  • Adds support for running the NGINX Plus data plane as a particular user.
  • Adds support for retrieving an NGINX Plus certificate and key via the REST API. You can download the certificate and key as a JSON or gzip file.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

  • The rewrite directive is not associated with a particular URI or location block in the generated configuration (10473)
  • SELinux settings can prevent statistics flow from agents (10342)
  • Code sample in the Dev Portal may have unintended URL encodings (13908)
  • Configuration changes are not deployed to all instances when one instance is unavailable (14227)
  • Note for PUT Update An Authentication Provider is incorrect (14576)
  • Cannot log in to the NGINX Controller web interface using Firefox v77 (14800)
  • Controller Agent fails to download when deploying an instance to AWS on RHEL 7 (14886)
  • Upgrading Controller Agent returns an error that named instance already exists (14932)

Known Issues

Installation and Upgrade

  • NGINX Controller install script provides wrong commands for installing Docker CE on RHEL 7 (15666)

    When installing NGINX Controller on RHEL 7, the install script displays commands for installing Docker CE that are for are an older, unsupported version of Docker CE.

    Workaround:

    Before installing NGINX Controller on RHEL 7, you should install Docker CE 18.09 (recommended version).

  • Custom Dashboards must be re-created after upgrading from NGINX Controller 3.7 to 3.8 (15815)

    When upgrading from NGINX Controller 3.7 to 3.8, Custom Dashboards are not migrated. You must re-create any Custom Dashboards after completing the upgrade.

Analytics

  • Alerts set using NGINX Controller 3.6 or earlier need to be updated after upgrading to NGINX Controller 3.8 (14565)

    Alerts that are set using NGINX Controller 3.6 and earlier are not checked after upgrading to NGINX Controller 3.8; the “last checked” dates for these alerts are from before the upgrade.

    Workaround:

    To resume checking the alerts, go to the Analytics > Alerts page in the NGINX Controller user interface, then mute and unmute the alerts.

Apps and Services

  • Create App Component settings for URI rewrites and header modification display URIs for non-referenced gateways (16103)

    The Applicable URIs list for URI rewrites and header modifications on the Create App Component page shows both referenced and non-referenced gateway URIs to choose from. If the URI for a non-referenced gateway is selected, NGINX Controller returns an error like the following: “Applied URI: <non-referenced URI> is not a part of the ingress URI.”

    Workaround:

    When selecting a gateway URI from the Applicable URIs list, select only URIs for gateways that the component references.

Infrastructure

  • The Analyzer page reports 0 certificates when a cert is configured (14017)

    The Analyzer misevaluates SSL certificates associated with instances. The Analyzer page reports the number of certificates as 0 in the SSL block, even when certificates have been configured.

    Workaround:

    To see if your configuration is using a certificate:

    1. Open the NGINX Controller web interface and log in.
    2. From the NGINX Controller menu, select Services > Certs > Overview.
    3. On the Certs Overview page, select a certificate from the list. This opens a side panel where you can view the certificate’s details, including whether the certificate is associated with an app component and/or gateway.
  • Once registered with NGINX Controller, a datapath instance cannot be renamed (15835)

    The name of your data-path instance – that is, the location where you install the NGINX Controller Agent – is fixed and cannot be changed once the instance is registered with NGINX Controller.

    When upgrading the Controller Agent, the install script allows entering a different instance name; however, this name is meant for adding new instances only. Attempting to provide another instance name when upgrading the Controller Agent could cause metrics to be misreported.

    Workaround:

    When upgrading the Controller Agent you can omit the instance name, or if you include the name, you must use the existing instance name.

  • After upgrading to NGINX Controller 3.8, some statistics aren’t updated on the Graphs page (16064)

    After upgrading from NGINX Controller 3.7 to 3.8, you may notice that some statistics on the Instances Graphs page – in the web interface, select Instances > Graphs – aren’t updated for some configurations.

    Workaround:

    To resolve this issue, restart the nginx and controller-agent services on the NGINX Plus instance:

    • service nginx restart
    • service controller-agent restart

    After restarting these services, NGINX Controller should start reporting the statistics again.

Platform

  • Per-environment and guest user access is broken for upgrades from NGINX Controller 3.4 or earlier (15941)

    Due to a change to the permission structure in NGINX Controller 3.4, upgrades from pre-3.4 versions to post-3.4 versions – including upgrades to NGINX Controller 3.8 – can break access for guest users and for users with “per-environment” roles.

    Workaround:

    There are two workarounds for this issue, depending on your upgrade history and risk tolerance.

    • Option 1 (recommended): If you have not yet upgraded NGINX Controller from a pre-3.4 release, first upgrade to NGINX Controller 3.4, and then upgrade again to the latest release.

    • Option 2: If you have already upgraded from a pre-3.4 release to a post-3.4 release without upgrading to 3.4 in the interim, you can only work around this issue by elevating the permissions of your guest users to regular users.

      Warning: This change involves a significant elevation of privilege and should be undertaken only with careful consideration. The regular user role has all the same admin role permissions except for the ability to promote new admins. If this change is acceptable for your installation – for any guest role users and any per-environment users with the guest role – these users should be edited by the default admin account to remove the guest role and replace it with the user role.

Documentation

  • NGINX Controller Agent troubleshooting guide incorrectly refers to stub_status module (16028)

    In the NGINX Controller troubleshooting guide, which you can find at https://<Controller-FQDN>/docs/support/troubleshooting-controller, some of the information in the section covering metrics is incorrect.

    See the following steps and corrections:

    • Step 4. Make sure the Controller Agent is using the same user ID as the NGINX worker processes. The default user ID is nginx.

      Correction: The default user ID for the NGINX Controller Agent is root.

    • Step 8. Make sure stub_status is configured, and the stub_status module is included in the NGINX build. To check, run the following command:

      nginx -V
      

      The output should mention --with-http_stub_status_module.

      Correction: The NGINX Controller Agent gathers metrics via the NGINX Plus API, not via the stub_status module.

    Workaround:

    When troubleshooting issues with metrics, instead of checking that stub_status is enabled, you should verify that nginx api is enabled.

  • The applies-to version list at the bottom of some docs doesn’t include NGINX Controller 3.8 (16795)

    At the bottom of the NGINX Controller docs, there’s a list of NGINX Controller versions that each doc applies to. For some docs installed with NGINX Controller 3.8, this list wasn’t updated. All of the docs installed with NGINX Controller 3.8 apply to that version.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21
  • NGINX Plus R22

NGINX Controller Version 3.7.0

July 30, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.7.0, in the following categories:

Updates

NGINX Controller 3.7.0 includes the following updates:

  • Bug fixes and improvements.
  • Bundles Kubernetes dependencies with the NGINX Controller installer, making installations more straightforward and reliable.
  • Disables port 80 for the NGINX Controller API gateway. Previously, NGINX Controller used port 80 to make initial redirects to HTTPS on port 443. Now, port 80 is disabled for the API gateway, and redirects aren’t supported. Port 443 is required to explicitly add https:// when pasting the FQDN in the address of the browser or when making API calls. Note: When you upgrade NGINX Controller, you also need to upgrade the NGINX Controller Agent software on each monitored NGINX Plus Instance.
  • Updates the Controller Agent to use golang instead of Python.
  • Adds Role Groups. Role Groups act as a collection of Roles that you can manage as a single resource. Role Groups can be mapped only to external auth provider groups; they cannot be assigned to local users.
  • Adds unique, individual salt to secure account passwords. Note: We recommend that users update their existing passwords to take full advantage of this enhancement.
  • Adds “Top X” functionality to limit the number of returned series in metrics queries to the top X data of probes. Multiple series that don’t fit into a limit are collapsed into a single series.
  • Adds a ‘dimension’ query parameter to the metrics API, which can be used to define the dimensions list to return for each metric series.
  • Adds a filtering option for refining the data that’s forwarded to Splunk using the data forwarder.
  • Introduces a new user interface for the Dev Portal. Create and manage beautiful, easy-to-use API reference documentation to support your published APIs.
  • Adds support for importing OpenAPI v2 specs (OAS) for your APIs.
  • Adds Total Latency metrics to the Environments, Apps, and App Components overview pages.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

  • Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)
  • Certificates imported out of order using drag-and-drop cause an error when creating a cert bundle (10761)
  • Importing API specs fails when done in the web interface (13501)
  • Cannot log in to NGINX Controller when admin email address contains uppercase letters (13775)

Known Issues

General

  • Cannot log in to the NGINX Controller web interface using Firefox v77 (14800)

    When trying to log in to the NGINX Controller web interface using Firefox v77, the following error occurs: SyntaxError: invalid regexp group IdentityProvider.form.tsx:364:17 <anonymous> IdentityProvider.form.tsx:364 Webpack 28.

    Workaround:

    Upgrade to Firefox v78.

API Management

  • Code sample in the Dev Portal may have unintended URL encodings (13908)

    In some cases, the example code snippet in the Dev Portal is URL encoded when the API has path parameters. For example, the code snippet might look similar to the following:

    curl --request GET --url http://sportsapp.example.com/api/f1/drivers/%7BdriverId%7D
    

    Workaround:

    You must remove the URL encoding before running the example code snippet, like so:

    curl --request GET --url http://sportsapp.example.com/api/f1/drivers/{driverId}
    
  • Known issues creating a Dev Portal using the NGINX Controller API (14385)

    • When passing a partial or incomplete theme customization payload to Dev Portal CRUD operations, the Dev Portal uses the missing settings’ default values. To customize fully, include the complete payload.
    • The Dev Portal supports only a curated list of fonts. You can see the list of accepted fonts while configuring a Dev Portal theme using the NGINX Controller web interface. If an invalid font value is submitted using the API, there’s no validation on the font value, and the default font is used instead.

    Workaround:

    Use the NGINX Controller web interface to construct a valid payload for creating and customizing the Dev Portal.

  • Error status of Published API component affects publishing to Dev Portal (14452)

    The Dev Portal can be created and associated with a Published API at the same time. However, if the Published API is in an error state when the Dev Portal is created, an error is issued that incorrectly says that creating the Dev Portal failed. Then if the Dev Portal is resubmitted, the association fails with an “object exists” error. Furthermore, if the Published API is in an error state, publishing documentation to the Dev Portal for that API also fails.

    Workaround:

    Check the component status for the Published API before creating and associating a Dev Portal for it.

  • Selecting between recent API Definitions and Dev Portals doesn’t refresh the webpage (14602)

    On the APIs page, selecting between items in the Recent API Definitions and Dev Portals quick list doesn’t refresh the page. Whichever item is displayed first is persistent.

    Workaround:

    To view specific API Definitions, select APIs > API Definitions. Then from the API Definitions list page, choose the API Definition that you want to view. Likewise, to view specific Dev Portals, select APIs > Dev Portals. Then from the Dev Portal list page, choose the Dev Portal that you want to view.

Infrastructure

  • Configuration changes are not deployed to all instances when one instance is unavailable (14227)

    In multi-instance environments, when a configuration change cannot be deployed to one instance, such as when an instance is not available, the configuration is not deployed to any of the other instances.

  • Controller Agent fails to download when deploying an instance to AWS on RHEL 7 (14886)

    When deploying an instance to Amazon Web Services (AWS) on RHEL 7, the installation fails, and the following error is logged to /var/log/agent_install.log: “Failed to download the install script for the agent.”

    Workaround:

    • Ensure that ports 443 and 8443 are open between NGINX Controller and the network where the instance is being deployed.
    • Verify that NGINX Controller can be accessed from the instance using the FQDN that was provided when NGINX Controller was installed
    • Ensure that the image referenced in the instance_template has a cURL version of 7.32 or newer.
  • Upgrading Controller Agent returns an error that named instance already exists (14932)

    When Upgrading the Controller Agent, the install script warns that the named instance already exists and says to choose another name or remove the existing instance.

    Workaround:

    This error can be ignored for Controller Agent upgrades.

Documentation

  • Note for PUT Update An Authentication Provider is incorrect (14576)

    In the NGINX Controller API reference guide, the description for the PUT /platform/auth/providers/{providerName} endpoint incorrectly says, “You must provide the full request body, including all configuration options, in a PUT request.”

    As long as the required parameters are included in the PUT request, the update works.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21
  • NGINX Plus R22

NGINX Controller Version 3.6.0

July 1, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.6.0, in the following categories:

Updates

NGINX Controller 3.6.0 includes the following updates:

  • Bug fixes and improvements.

  • Adds external authentication using Active Directory (beta).

    Note: Active Directory users are given user-level access to NGINX Controller in this beta implementation.

  • Adds ability to create orchestrated Instances on AWS using Instance Templates.

  • Adds metrics forwarding for Splunk (beta).

  • Adds ability to secure published APIs by configuring authentication, conditional access, and rate limiting at the Component level.

  • Adds ability to configure DNS Service Discovery at the Component level.

  • Introduces a new user interface for Identity Providers.

  • Adds ./opt/nginx-controller/helper.sh repository-cred for getting the repo key and cert, needed for installing NGINX Plus.

  • Changes the default security option for Instance registration. Registration of NGINX Plus Instances is now performed over a secure connection by default. A new option – Allow insecure server connections to NGINX Controller using TLS – is available in the user interface. This option allows you to use self-signed certificates with the Controller Agent. For security purposes, we recommend that you secure the Controller Agent with signed certificates when possible.

End of Support: Debian 8 (i386, x86_64) is unsupported as of June 30, 2020. As such, NGINX Controller Agent support for Debian 8 will end with NGINX Controller version 3.7. End of Support means that NGINX will not fix bugs related to Debian 8 in NGINX Controller 3.7 and later.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for each issue.

  • Command to install Controller Agent includes --no-check-certificate by default (4166)
    • Solution Article: K31150658 | CVE-2020-5909
  • NATS services do not require authentication (6404)
    • Solution Article: K59209532 | CVE-2020-5910
  • On Debian and Ubuntu, Kubernetes packages are downloaded over plaintext HTTP (11771)
    • Solution Article: K84084843 | CVE-2020-5911

Additional Fixes

  • REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)
  • When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)
  • Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)
  • Component URI rewrites that are added and removed using the user interface cause a rewrite error (12527)
  • The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)
  • Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)
  • API resources with POST method must be defined using the NGINX Controller API (12652)

Known Issues

Installation and Upgrade

  • Controller Agent warns that Instance name already exists during upgrade (13350)

    When upgrading the Controller Agent, the install.sh script issues a warning that the Instance name already exists. This validation is a safeguard for new installations. You can safely disregard this warning for upgrades.

  • Cannot log in to NGINX Controller when admin email address contains uppercase letters (13775)

    The installer for NGINX Controller allows users to create admin accounts with email addresses that contain uppercase letters. However, NGINX Controller expects emails to be only lowercase and prevents admin accounts with differently formatted email addresses from logging in.

    Workaround:

    You can reinstall NGINX Controller and enter a lowercase email address for the admin account. If that doesn’t work, contact NGINX Support for assistance.

API Management

  • Published APIs do not go into an error state when the Instance they are using is deleted (10616)

    If you delete the last Instance serving a published API, and then edit the published API, the published API gets stuck in the Configuring state. Attempting to delete the published API causes the published API to get stuck in a Deleting state.

  • Rate-limit policies are lost after upgrading to NGINX Controller v3.6 (12962)

    Existing rate-limiting policies applied to published APIs are lost when upgrading to NGINX Controller 3.6. The policies must be reconfigured at the Component level following the upgrade.

  • JWK files need to be resubmitted after upgrading from NGINX Controller v3.4 to v3.6 (13289)

    Existing JWK files applied to Identity Providers are lost when upgrading from v3.4 to v3.6 and must be resubmitted following the upgrade.

  • Importing API specs fails when done in the web interface (13501)

    Importing API specs from within the NGINX Controller user interface fails because of how the interface parses the text.

    Workaround:

    Import API specs using the NGINX Controller REST API.

Infrastructure

  • Restarting the Controller Agent service leaves behind PID file with errors logged to agent.log (13390)

    When the Controller Agent service is restarted, the controller-agent.pid file is left behind, and errors about the PID file are logged in agent.log.

    Workaround:

    Manually remove /var/run/controller-agent/controller-agent.pid, then restart the Controller Agent service again. The Controller Agent will work normally after it restarts.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21
  • NGINX Plus R22

NGINX Controller Version 3.5.0

June 9, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.5.0, in the following categories:

Updates

NGINX Controller 3.5.0 includes the following updates:

  • Bug fixes and improvements.
  • Adds support for NGINX Controller on RHEL 7.
  • Deprecates support for NGINX Controller Agent on CentOS 6.
  • NGINX Controller requires the yum-plugin-versionlock package on RHEL and CentOS.
  • Adds Gateway dimension to the metrics API.
  • Adds proxy parameters for Gateways and Components to manage retry attempts on error conditions.
  • Adds socket parameters for Gateways and Components to manage incoming TCP connections for the data plane.
  • Introduces a new user interface and declarative API for API Management.
  • Adds support for importing an OpenAPI specification when creating an API Definition.
  • Adds versioning for your API Definitions.
  • The Global Settings endpoint supports configuring API Gateway certificates and database settings.
  • Group your NGINX Instances by their physical locations using the Locations setting.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

  • The NGINX Controller web interface doesn’t have sufficient protection against a Cross-Site Request Forgery (CSRF) attack. (4882, 9721)
    • Solution Article: K31044532 | CVE-2020-5900
  • The account recovery code for NGINX Controller that’s sent in email is not secure (11092)
    • Solution Article: K25434422 | CVE-2020-5899
  • Undisclosed API endpoints may allow for a reflected Cross-Site Scripting (XSS) attack (12021)
    • Solution Article: K43520321 | CVE-2020-5901

Additional Fixes

  • Cannot delete an instance that doesn’t have NGINX Plus installed (7704)
  • NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)
  • Cannot change SMTP password using the platform/global endpoint (11049)
  • NGINX Controller v3 does not support RHEL 7 (11282)
  • Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)
  • Security section appears in the documentation but has no content (11294)

Known Issues

Installation and Upgrade

  • Package error when installing nginx-plus-module-njs on CentOS 7 (12638)

    When installing nginx-plus-module-njs on CentOS 7, the installer returns a package error.

    Workaround:

    1. Uninstall the existing nginx-plus-module-metrics module:

      sudo yum remove -y nginx-plus-module-metrics
      
    2. Install the latest version of nginx-plus-module-njs:

      sudo yum install -y nginx-plus-module-njs
      
    3. Install the nginx-plus-module-metrics module, specifying the version as 1.19:

      sudo yum install -y nginx-plus-module-metrics-1.19*
      

    Verification:

    1. To confirm that the correct versions are installed, run the following command:

      yum list installed | egrep 'nginx|metrics'
      

      The expected output looks similar the following:

      avrd-libs.x86_64 0.15.5_1850598.release_3_5-1 @nginx-controller-metrics
      nginx-controller-agent.x86_64 3.5.0-10484161.el7 @nginx-controller
      nginx-plus.x86_64 22-1.el7.ngx @nginx-plus
      nginx-plus-module-metrics.x86_64 1.19.0-1850598.el7.ngx @nginx-controller-metrics
      nginx-plus-module-njs.x86_64 22+0.4.1-1.el7.ngx @nginx-plus
      

API Management

  • The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)

    The URIs for a Published API use the PREFIX match method, regardless of the resource match method that was specified in the API Definition.

  • API resources with POST method must be defined using the NGINX Controller API (12652)

    When defining an API Definition version with a resource that uses the POST method and specifies an example request, the generated API spec in the web interface is invalid.

    Workaround:

    Use the NGINX Controller API to define API Definition version resources that use the POST method and contain an example request in the API documentation.

Apps and Services

  • Component URI rewrites that are added and removed using the web interface cause a rewrite error (12527)

    If a URI rewrite for a Component is added and then deleted using the NGINX Controller web interface, the API spec that’s sent to NGINX Controller includes an empty programmability section. This results in an Error status for the Component, and an error similar to the following is returned: "Error: Arguments of rewrite : incomingPattern or rewritePattern is missing."

    Workaround:

    • Option 1 (Recommended): Send a PUT request to the Components endpoint that excludes the programmability section.
    • Option 2: Configure a placeholder rewrite that writes from a pattern to a pattern. Using a pattern that doesn’t match any request is best.

Infrastructure

  • Controller Agent installation doesn’t warn about duplicate instance names until an Instance has been registered with NGINX Controller (12370)

    The Controller Agent installation script typically fails with a warning if the provided instance name already exists for the specified location. However, a race condition occurs if the installation script is run on two instances at the same time with the same name for the same location. When this happens, the failed installation doesn’t issue a warning, but only one instance is added to NGINX Controller.

    Workaround:

    No workaround is necessary. The behavior is the same as if the installation script displayed the warning.

  • Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)

    When the nginx-controller-agent package is removed on an NGINX Plus instance using apt-get-purge, the /etc/controller-agent directory and its contents are deleted. This directory is where NGINX Controller places certificates when NGINX Controller is used to configure and manage NGINX Plus instances. When this directory is deleted, the NGINX Plus instance becomes unresponsive and the NGINX service fails to start.

Platform

  • Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)

    When a database certificate is added that uses an IP address instead of an FQDN, or if the certificate Common Name (CN) is invalid, the NGINX Controller database services reject the invalid cert and refuse database connections. In this case, the system becomes unresponsive until a valid cert is provided.

    Workaround:

    Use the hostname instead of an IP address, or use a certificate where the IP address is mentioned in Subject Alternative Name (SAN) as DNS type.

  • NGINX Controller Agent and AVRD do not work on 32-bit distributions (12605)

    The Analytics, Visibility, and Reporting daemon (AVRD) is compiled as a 64-bit app and does not run on 32-bit distributions. Configuration Management using the Controller Agent is possible on 32-bit distributions; however, metrics are not reported.

    Workaround:

    Install the NGINX Controller Agent and AVRD on a 64-bit distribution.

    Important: NGINX Controller support for 32-bit distributions will be deprecated in a future release.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21
  • NGINX Plus R22

NGINX Controller Version 3.4.0

May 6, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.4.0, in the following categories:

Updates

NGINX Controller 3.4.0 includes the following updates:

  • Bug fixes and improvements.
  • Improved Controller Agent error messages that make it easier to troubleshoot connectivity issues during installation.
  • Write permissions for the platform/global endpoint are limited only to the Admin role now, to safeguard against changes to critical server and database configurations.
  • Audit events are exposed using the /events endpoint.
  • Write permissions for the platform/global endpoint are limited only to the Admin role now, to safeguard against changes to critical server and database configurations.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

  • Malformed messages lead to segmentation fault of the Analytics, Visibility, and Reporting daemon (AVRD) (9068)
    • Solution Article: K95120415 | CVE-2020-5895
  • The Analytics, Visibility, and Reporting daemon (AVRD) is world-readable and writeable (9067)
    • Solution Article: K95120415 | CVE-2020-5895
  • The NGINX Controller webserver does not invalidate the the server-side session token when users log out (8576)
    • Solution Article: K13028514 | CVE-2020-5894

Additional Fixes

  • API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

Known Issues

Installation and Upgrade

  • Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)

    When the conditions noted below are met, the Identity Provider API keys are not migrated and must be re-entered.

    Conditions:

    This issue applies when the following conditions are true:

    • You are running NGINX Controller v3.0.
    • You are running NGINX Controller v3.2, which you previously upgraded from NGINX Controller v3.0.
    • You are running NGINX Controller v3.3, which you previously upgraded from NGINX Controller v3.0.

    Workaround:

    Step 1: Copy and save your API Identity Profile keys

    Before upgrading, you must complete the following steps to copy and save any API Identity Profile keys that you have.

    1. Open the NGINX Controller web interface and log in.
    2. Select the NGINX Controller menu icon, then select Services.
    3. On the Services menu, select APIs.
    4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
    5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
    6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
    7. Copy the Identity Provider API key value and save it locally. You’ll need to re-enter this value after you upgrade.

    You can now complete the upgrade to NGINX Controller v3.4.

    Step 2: Restore your API Identity Profile keys

    After you’ve upgraded to NGINX Controller v3.4, you can restore the API Identity profile keys that you saved.

    1. Open the NGINX Controller web interface and log in.
    2. Select the NGINX Controller menu icon, then select Services.
    3. On the Services menu, select APIs.
    4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
    5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
    6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
    7. Paste the copied Identity Provider API key that you saved in the previous procedure into the key field. Then, click Save.
  • NGINX Controller v3 does not support RHEL 7 (11282)

    NGINX Controller v3 does not support Red Hat Enterprise Linux (RHEL) v7. The NGINX Controller installation fails during the Docker version check. Manually updating the Docker version on RHEL 7 does not solve the problem.

Apps and Services

  • Certificates imported out of order using drag-and-drop cause an error when creating a cert bundle (10761)

    When certificates are imported using the drag-and-drop feature in the NGINX Controller web interface, the system expects the first certificate that’s added to be the public certificate. Additional certificates that are added are treated as intermediate CA certs.

    If the certificates aren’t added in order, with the public certificate added first, the resulting API request has transposed values for the publicCert and caCert parameters.

    Workaround:

    When adding certificates using drag-and-drop, always drag the public cert file in first, followed by any additional CA certificates. The private key can be dragged in either before or after the certificates. Alternatively, you can choose to copy and paste the PEM text into the corresponding fields.

Platform

  • Cannot change SMTP password using the platform/global endpoint (11049)

    Sending the SMTP password in a PATCH request to the platform/global endpoint has no effect.

    Workaround:

    To change the SMTP password, you can use the help.sh script that’s located in /opt/nginx-controller/ on the NGINX Controller host. For instructions, see the documentation that’s installed with NGINX Controller: https://<Controller-FQDN>/docs/platform/using-helper-script/#update-smtp-settings.

Documentation

  • Security section appears in the documentation but has no content (11294)

    In NGINX Controller v3.4, a “Security” panel appears in the “Services” section of the onboard documentation. If you click on the Security panel, you will not see any content listed because there is no security documentation for this release. Users should disregard the “Security” panel in the documentation for this release.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21

NGINX Controller Version 3.3.0

April 09, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.3.0, in the following categories:

Updates

NGINX Controller 3.3.0 includes the following updates:

  • Bug fixes and improvements.
  • Adds support for certificate chains for TLS connections.
  • Adds an API catalog for the Metrics API.
  • Adds new dimensions for filtering metrics on the Application Summary page.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

  • The NGINX Controller Agent installer script ‘install.sh’ uses HTTP instead of HTTPS to check and install packages (8587)
    • Solution Article: K00958787 | CVE-2020-5867
  • The helper.sh script, which is used optionally in NGINX Controller to change settings, uses sensitive items as command-line arguments (7436)
    • Solution Article: K11922628 | CVE-2020-5866
  • NGINX Controller uses insecure connection for communications with PostgreSQL database by default (6509)
    • Solution Article: K21009022 | CVE-2020-5865

Additional Fixes

  • Creation of eventually consistent objects returns 201 instead of 202 (7052)
  • Can’t delete a configured gateway when there is a gateway in an error state (7058)
  • Dependencies Missing from Open Source Dependencies Report (7404)
  • Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)
  • Intermediate certificates not pushed to NGINX Plus instance (7500)
  • Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)
  • Agent warns about failed connection to receiver after update (9152)
  • Certificate expiration shows expired warning and expiration date that doesn’t match the final certificate (9167)
  • Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)
  • Login endpoint has inaccurate session information in the API Reference Guide (9482)
  • A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

Known Issues

Apps and Services

  • NGINX Controller web interface shows “Error fetching apps” after a restart (10013)

    In some cases, after NGINX Controller restarts, you might see an error in the web interface that says, “Error fetching apps: Network Error.”

    Workaround:

    Log out of the web interface and then log back in.

  • Gateways configured with duplicate ingress URIs error out with condition of “config” (10425)

    When upserting a Gateway, if the ingress URI (desiredState.ingress.uris) is a duplicate of another Gateway’s ingress URI, the upsert errors out as expected; however, the condition type (state.conditions.type) that’s reported is simply config, instead of an explanation for the condition that’s occurred.

    "state": {
        "conditions": [
            {
                "type": "config" 
            }
        ],
        "selfConfigState": {
            "configured": 0,
            "configuring": 0,
            "deleting": 0,
            "error": 1,
            "isConfigured": false,
            "isConfiguring": false,
            "isDeleting": false,
            "isError": true,
            "total": 1
        }
    }
    

    Workaround:

    To resolve the error, remove the Gateway with the duplicate ingress URI that’s in the error state. You can do this by issuing a PUT request and applying a new config with a unique ingress URI, or you can issue a DELETE request to remove the Gateway with the duplicate ingress URI.

  • Component fails to configure when its Gateway URI collides with another Gateway’s URI (10430)

    In situations where a Component for one Gateway uses the FQDN of a different Gateway on the same host, NGINX Controller doesn’t push the config to NGINX Plus.

    Scenario

    1. Create a Gateway for foo.com with the hostname http://foo.com, and deploy to NGINX Plus ‘Instance 2’.
    2. Create a Component for foo.com with the URI http://foo.com/, and associate the Component with the Gateway foo.com and assign a workload.
    3. Create a Gateway for bar.com with the hostname http://bar.com, and deploy to NGINX Plus ‘Instance 2’.
    4. Create a Component for bar.com with the URI http://foo.com/baz, and associate the Component with the Gateway bar.com and assign a workload.

    In this scenario, the configuration should expose the location for /baz. However, although the Component is reported as configured, nothing happens.

  • App Services API Known Issues (10473)

    The App Services API has known issues for certain attributes and objects, in particular:

    • Socket parameters defined in the API should not be used at all. This includes the receivBufferSize and sendBufferSize parameters that are common to Gateway ingress and Component ingress.
    • Session persistence works only for SessionPersistenceCookie. Using SessionPersistenceRoute or SessionPersistenceCookieLearn results in configuration errors.
    • When configuring workload monitoring, and when using the API to check headers in the response, it’s necessary to check for an exact match for the header. Matching with a regular expression or checking only for the existence of a header does not work.
    • A Load Balancing method of HASH won’t work if it contains a variable such as $request_URI for the userKey.

    The existing URI Rewrite capability in the App Services API has the following limitations:

    • The Rewrite directive is not associated with a particular URI or location block in the generated configuration.
    • The break flag is not supported.

Analytics

  • SELinux settings can prevent statistics flow from agents (10342)

    Depending on the SELinux settings, an NGINX instance might not have permission to submit statistics to your NGINX Controller.

    If you have a properly configured environment, application, and component for which no metrics are being passed to NGINX Controller after more than 30 seconds of traffic have transpired, try the suggested workaround steps.

    See the NGINX blog post Using NGINX and NGINX Plus with SELinux for additional general guidance.

    Workaround:

    1. On the instance(s) associated with your Gateway, run the following command as root:

      # grep nginx /var/log/audit/audit.log | audit2allow -M nginx`
      
    2. Then, run this command:

      # semodule -i nginx.pp
      

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20
  • NGINX Plus R21

NGINX Controller Version 3.2.0

March 17, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.2.0, in the following categories:

Updates

NGINX Controller 3.2.0 includes the following updates:

  • Bug fixes and improvements.
  • API definition URIs support substitutions. When creating or editing a URI within an API Definition, you can enter substitution patterns by using the {} notation.
  • Adds support for tagging instances.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

  • API allows for unauthenticated account creation (9428)
    • Solution Article: K14631834 | CVE-2020-5863
  • NGINX Controller Agent uses insecure connection for communications between NGINX Controller and NGINX Plus instances by default (5899)
    • Solution Article: K27205552 | CVE-2020-5864

Additional Fixes

  • Stack trace error occurs when accessing the Login credentials schema in the API reference docs. (6409)
  • JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)
  • Links to metrics descriptions from user interface point to the metrics reference landing page (7255)
  • Usage of API key authentication policies requires the njs module on data-plane instances (7298)
  • Omitting URI when defining a component may result in unexpected behavior (7308)
  • Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

Known Issues

Apps and Services

  • Certificate expiration shows expired warning and expiration date that doesn’t match the end-entity certificate (9167)

    A certificate might appear as expired on the certificate detail page, although the Valid To date is in the future. This happens because the summary status and expiration date are calculated based on the entire certificate chain from which the certificate is derived. So, if you have an intermediate CA cert in your chain, the expiration summary status and summary expiration date may reflect the status/date of any intermediate certificate. If you see such a status or expiration date, you should check the status of your entire certificate chain, including that of any intermediate CA.

  • A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

    A high volume of Create, Update, and Delete (CUD) operations – for example, more than 90 – performed on Gateways and Components could create a condition in which NGINX Controller becomes unresponsive.

    NGINX Controller limits queued operations at 100 and then returns an HTTP 429 error: “Too many requests; wait, and try again.” If, subsequently, additional CUD operations are submitted at a rapid rate (more than 1 per second), the queue may be overrun, creating a condition where no further changes can be processed without involvement from NGINX Support.

    Workaround:

    To avoid this situation, limit the amount of API Create, Update, and Delete (CUD) operations sent to the Component and Gateway endpoints to fewer than 100 open actions. The queue should clear 1 action per 7 seconds, on average.

Infrastructure

  • Agent warns about failed connection to receiver after update (9152)

    After updating NGINX Controller and the Controller Agent to v3.2.0, the system warns that the Agent “Failed to connect to the receiver.” However, if you look in the agent.log file, the Agent is logging communication with the receiver, as expected.

    This warning occurs when a TLS certificate signed with a custom CA is used for the NGINX Controller. Before v3.2.0, the Controller Agent did not verify server certificates.

    To determine the status of the Agent, perform the following on the Agent host:

    • Check the log file /var/log/nginx-controller/agent.log for errors.

    • Check the status of the Controller Agent service:

      systemctl status controller-agent

      The output should include the status running.

  • Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)

    If a TLS certificate for NGINX Controller is signed with a custom CA, the Agent fails to verify the cert and returns an error similar to the following:

    2020-03-09 20:38:28,803 [5089] MainThread failed POST "https://<fqdn>:8443/1.4/<uuid>/agent/",
    exception: "HTTPSConnectionPool(host='<fqdn>', port=8443): Max retries exceeded with url: /1.4/<uuid>/agent/
    (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))"
    

    Workaround:

    1. Place the CA on the Agent host.

    2. On the Agent host, edit the /etc/controller-agent/agent.conf file. In the cloud directive section, make the following changes:

      • Add or set the requests_ca_bundle option to point to the CA file.
      • Set verify_ssl_cert to True. (Note: If this setting is set to False, then server certificate validation is disabled, which we don’t recommend.)
      [cloud]
      ...
      verify_ssl_cert = True
      requests_ca_bundle = <path-to-ca-bundle-file>
      
    3. Save the changes to the agent.conf file.

    4. Restart the agent:

      systemctl restart controller-agent

    5. Verify that the Agent can connect to the NGINX Controller host:

      systemctl status controller-agent

      The output should include the status running. You can also check the log file /var/log/nginx-controller/agent.log for errors.

Documentation

  • Login endpoint has inaccurate session information in the API Reference Guide (9482)

    The API reference documentation for the Login endpoint describes new session behavior that hasn’t been implemented yet. At this time, the default maximum session lifetime is 8 hours. Once the session lifetime limit is exceeded, you must log in to obtain a new session token.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20

NGINX Controller Version 3.1.0

February 26, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.1.0, in the following categories:

Updates

NGINX Controller 3.1.0 includes the following updates:

  • Bug fixes and improvements.

  • The Components resource adds experimental strategyRef and waf placeholder objects.

    The strategyRef and waf objects for the Components resource are experimental placeholder objects. Including the strategyRef and/or waf blocks in the component specification should not enable additional security-related application traffic services, even if there are no errors. If you see the error “Security module not found on the NGINX instance,” you should remove these blocks.

    Note: The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension.

  • This release adds new app-centric metrics and a new summary page in the UI for viewing these metrics. You can use these app metrics to monitor and evaluate the condition of your apps.

Resolved Issues

This release includes the following fixes. Search by the issue ID – the number in parentheses – to locate the details for an issue.

  • Cannot add tags to users with the NGINX Controller web interface (7165)
  • Gateway accepts a relative path instead of a URI (7212)
  • A component with no URIs will result in an errored deployment (7311)
  • NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)
  • Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)
  • Referencing a non-existing identity provider for a component does not generate an error (7397)
  • GUI incorrectly shows default error log state as enabled (7447)
  • Component-based access logging format specification is non-functional (7646)
  • Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)
  • Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)
  • API definitions base path is not imported from OAS spec file (7671)
  • The API Reference documentation omits the “Get a Location” endpoint (7747)

Known Issues

Installation and Upgrade

  • Could not initialize a Kubernetes cluster when installing NGINX Controller on CentOS 7 (5863)

    There is an issue in CentOS 7 that creates a mismatch between the optimum Docker settings and optimum Kubernetes settings. This mismatch prevents Kubernetes from initializing during the installation of NGINX Controller on CentOS 7.

    The NGINX Controller installation fails after Docker and Kubernetes are installed, with the following error: “execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster.”

    Workaround:

    • Solution one:
      • Run sudo yum update. This updates the components to the latest version and patches the issue.
    • Solution two:
      1. Run controller-installer/install.sh and follow the instructions. Allow the installation to fail. This establishes Docker and Kubernetes.
      2. Stop Docker: sudo systemctl stop docker
      3. Edit /etc/docker/daemon.json and make the the following change: native.cgroupdriver=systemd to native.cgroupdriver=cgroupfs
      4. Start Docker: sudo systemctl start docker
      5. Reset the Kubernetes configuration: sudo kubeadm reset -f
      6. Run controller-installer/install.sh and follow the instructions. Rerunning the installation does not update the version of any of the packages on the machine.

Apps and Services

  • When committing a gateway with REGEX matchMethod, whose URI includes an explicit port and a path, the error message incorrectly states that the port is invalid (8452)

    In the context of configuring a gateway, only the protocol, domain, and optional port have any meaning. If the URI specified when configuring a REGEX matchMethod gateway includes a path, including a final / after the port, you will receive an error stating: Port value is invalid for hostname .*\\.example\\.com, where the hostname string corresponds to the data you provided for the domain.

    Workaround:

    • Specify your REGEX URI for a gateway as only the protocol, domain, and port with no trailing path.
  • REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)

    The logic for interpreting REGEX and REGEX_CASE_SENSITIVE match methods for REGEX-based Component URIs is reversed.

    REGEX-based Component URIs are converted into a case-insensitive location block in nginx.conf when the URI is specified as a relative URI (leads with a path, not a domain) and when the match method is specified as REGEX_CASE_SENSITIVE. Conversely, when the match method is REGEX, the REGEX-based Component URI is interpreted as case sensitive.

    If you need your REGEX-based Component URI to be interpreted as case sensitive, then you can specify the match method as REGEX for now. However, keep in mind that when this logic mismatch is fixed in a later release of NGINX Controller, you’ll need to revert the match method accordingly.

  • When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)

    In some cases, when communication between NGINX Controller and the Controller Agent is interrupted, an error similar to the following is returned: “Error: Failed to deploy to instance, Could not find the targeted NGINX running on the system None”. This error neglects to indicate the name of the system or systems for which the communication has failed.

    Workaround:

    • Communication difficulties can be persistent or temporary, depending on the nature of the failure. It is possible to see some more persistent communication failures by looking at the Infrastructure > Graphics pane in NGINX Controller. If the hostname and nginx-plus instance are down, these items will appear with a red background.

    • If all the systems and nginx-plus instances have with a white background, the communication problem could be temporary or intermittent, and won’t necessarily show in the interface.

      Things to check in order are:

      • Ensure the agent host/container is up
      • Ensure the controller-agent process is running
      • Ensure the nginx-plus process is running
      • Verify network communication between NGINX Controller and the Controller Agent on port 443.

Analytics

  • Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)

    When upgrading from NGINX Controller 3.0.0 to 3.1.0, the app-centric metrics are not sent from the NGINX instances to NGINX Controller until a new configuration push is performed.

    Workaround:

    • Perform a Put to an object on each NGINX instance to update the config.

Documentation

  • API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

    The API Management documentation states that the API Management module requires a dedicated NGINX Plus instance. This is no longer the case. As of v3.0, the API Management module no longer needs to run on a dedicated instance.

  • Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

    The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

    The following API actions are considered “experimental” features; use is not supported and functionality may change.

    • Create an Environment
    • Create an App
    • Create or update an App Component with the following properties defined in desiredState:
      • ingress.tcpKeepAlive
      • backend.workloadGroups.uris.failTimeout
      • backend.workloadGroups.uris.resolve
      • backend.workloadGroups.uris.route
      • backend.workloadGroups.uris.service
      • backend.workloadGroups.uris.slowStart
      • backend.workloadGroups.serviceDiscovery
      • backend.workloadGroups.serviceDiscovery.dnsResolver
      • backend.workloadGroups.serviceDiscovery.dnsResolver.uris
      • backend.preserveHostHeader
      • backend.queue
      • backend.httpVersion
      • backend.ntlmAuthentication
      • backend.persistentState
      • programmability.httpHttpsRedirect
      • programmability.uriRedirects
      • programmability.cookieModifications
      • programmability.requestHeaderModifications
      • programmability.responseHeaderModifications
      • security
      • security.waf
      • security.conditionalAuth
      • security.interceptWorkloadErrors
    • Create or update a Gateway with the following properties defined in desiredState:
      • ingress.headers
      • ingress.http2
      • ingress.spdy
      • ingress.proxyProtocol
      • ingress.setFib
      • ingress.fastOpen
      • ingress.backlog
      • ingress.acceptFilter
      • ingress.deferred
      • ingress.isIpv6Only
      • ingress.reusePort
      • ingress.notFoundStatusCode
  • Stack trace errors occur in the API reference docs (8481)

    In NGINX Controller, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/.

    For a number of elements, clicking on an object to view the nested content will result in a ReDoc stack trace error.

    Workaround:

    Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.

  • Dependencies Missing from Open Source Dependencies Report (7404)

    In NGINX Controller v3.1, the following open source dependencies were omitted from the open source acknowledgements PDF:

    • nats version 2.1.0 (https://github.com/nats-io/nats.c)
    • protobuf version 3.10.1 (https://github.com/protocolbuffers/protobuf)
    • protobuf-c version 1.3.2 (https://github.com/protobuf-c/protobuf-c)

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20

NGINX Controller Version 3.0.0

January 20, 2020

These release notes provide general information and describe known issues for NGINX Controller version 3.0.0, in the following categories:

Updates

  • NGINX Controller Upgrade not supported in v3.0.0

    For NGINX Controller v3.0.0, upgrade from previous versions is not supported. Breaking changes were introduced between v2.9 and v3.0.0; as such, the upgrade.sh script is not present in v3.0.0.

  • New look for the web interface and docs

    The NGINX Controller v3.0.0 web interface has a brand-new look that we’re excited to share. Intuitive forms help guide you through your tasks. You’ll notice that features are arranged into four categories to match your workflow:

    • Analytics - View your application health score, alerts, and events
    • Services: Define your apps and their services
    • Infrastructure - Manage your NGINX Plus instances
    • Platform - Manage your NGINX Controller settings and user accounts

    The documentation also has a new look. There’s an API reference guide that documents the REST API endpoints. If you use the web interface, the related documentation is organized by the same categories that you see in the UI so that you can quickly find the topics related to your tasks.

  • Generate Support Package by Using the “helper” Script

    New in NGINX Controller v3.0.0, you can pass the supportpkg argument to the helper.sh script to create a support package. The script gathers NGINX Controller configuration files, logs, and system command output into a tarball. You can use this information to diagnose NGINX Controller issues and/or send the tarball to NGINX Support for troubleshooting assistance. Certs are excluded from the support package.

    Example usage:

    [email protected]:/var/tmp$ /opt/nginx-controller/helper.sh supportpkg
    Gathering System Diagnostics: Please wait ...
    Finished: /var/tmp/supportpkg-20191231T053348MST.tar.gz
    
  • Events API

    This release adds an API for the events that were present in Controller version 2.x.

  • Metrics API

    This release adds an API for the metrics that were present in Controller version 2.x.

Known Issues

API Management

  • Data displayed by the user interface does not match that returned by the REST API for some API Management endpoints (7117)

    When working with API Definitions and Published APIs in the API Management module, the following mismatches between the NGINX Controller user interface and the REST API have been reported. These are dependent on where the user created or modified the affected objects.

    • Identity Provider - API Key: When you create an Identity Provider that uses API Key authentication by using the REST API, the name of the Identity Provider group will not display in the user interface. This behavior resolves if you modify the Identity Provider resource by using the REST API.
    • Identity Provider Client - API Key: When you create an Identity Provider Client that uses API Key authentication by using the user interface, you will receive a “404 - Not Found” error in response to an HTTP GET request for the resource.
    • Published API: When you create a Published API by using the user interface, the Environment, Application, Gateway, Policy, and Routes will be missing from the response to an HTTP GET request for the resource.
    • API Definition: When you create an API Definition by using the REST API, the name of the API Definition is not displayed in the user interface. When you create an API Definition by using the user interface, the URIs will be missing from the response to an HTTP GET request for the resource.
  • Usage of API key authentication policies requires the njs module on data-plane instances (7298)

    In the NGINX Controller documentation for the API Management module, we state that “You must install the njs JavaScript module on each NGINX Plus instance to enable API key hashing” when using API key authentication for an Identity Provider. This information is not accurate.

    You must install the njs module on all NGINX Plus instances if you want to use API key authentication for any element of NGINX Controller (not just for API Management). This enables cryptographically-protected storage of API keys in the NGINX Plus configuration.

    If you do not install the njs module and use API key authentication, whether for API Management or elsewhere, the system may experience errors that are not reported in the user interface. See the NGINX Admin Guide for njs installation instructions.

  • API definitions base path is not imported from OAS spec file (7671)

    After you use the API Management REST API to create an API Definitions from an OpenAPI 3.0 spec file, the base path is empty.

    Workaround:

    • You must manually edit the API Definition and provide the base path for API definitions that are created from imported OAS spec files.
  • JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)

    Within a single Published API, configuring both an API key authentication policy and a JWT authentication policy results in an error, and you will not be able to publish.

    Workaround:

    • If you’re using the NGINX Controller user interface, only add one type of authentication policy (API Key or JWT) per Published API.
    • If you need to use an API key policy and a JWT policy within a single Published API, you can configure those policies using the external API, on condition that the keys are presented in the header or header/bearer, respectively.

Installation and Upgrade

  • NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)

    When installing NGINX Controller, the install.sh script misidentifies the Docker version that’s installed as 19.03.5. NGINX Controller actually installs docker-ce 18.09 and docker-ce-cli 19.03.5. These versions are supported.

    Note: The Docker Command Line Interface (docker-ce-cli) is not an NGINX Controller requirement. Rather, it’s an interface for the administrator to use when working with the Docker Daemon (docker-ce).

Apps and Services

  • Creation of eventually consistent objects returns 201 instead of 202 (7052)

    Component and gateway objects are considered eventually consistent and will not be immediately configured when created. Currently, these endpoints always return a 201 HTTP code (status created) after a PUT (create) or POST. A 202 HTTP code (status accepted) should be expected instead.

  • User interface cannot be used to correct some error cases from direct API calls (7247)

    A direct API call can result in an errored state that cannot be subsequently corrected via the user interface. For example, when creating a component, if the protocol in the workload URI is missing, the component is created and goes into the isError state. When the user interface is then used to try to add the protocol, the Publish button never becomes green to allow the change to be pushed.

    Workaround:

    • Via the API, use a PUT request to correct the error.
  • A component with no URIs will result in an errored deployment (7311)

    A component must have one or more URIs in its definition. If a component has no URIs, it will result in an error when being deployed, resulting in a failed deployment.

    Workaround:

    • Include one or more URIs in the definition of a component.
  • Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)

    When component-level access logging is enabled, either through the web interface or via the public API, publishing the configuration gets stuck in an inConfiguring state. It never advances to a Configured state.

    This condition will block updates to the relevant gateway until the underlying condition with the component is resolved.

    Workaround:

    • Do not enable component-level access logging.

      If you find yourself in this situation, you can perform a PUT using the public API – or republish via the web interface – for the relevant component with component-level access logging disabled. This will result in a configured system and will allow other updates to proceed correctly.

  • Component-based access logging format specification is non-functional (7646)

    When specified via the NGINX Controller public API (it is not available through the web interface), the format string for the component-level access log always errors out with an invalid string format error, regardless of the validity of the string that’s specified.

    Workaround:

    • For components to deploy and successfully be configured, avoid setting the component desiredState.logging.accessLog.format.
  • Gateway ingress URIs must specify either HTTP:// or HTTPS://, otherwise the gateway becomes stuck in Configuring state (7416)

    Gateway ingress URIs must specify either HTTP:// or HTTPS:// as the protocol for the ingress.uri. If the protocol is mistyped in a POST or PUT for a gateway, the gateway can become stuck in a Configuring state that never resolves nor issues an error.

    Workaround:

    • If you mistyped the ingress URI protocol, you can submit another PUT request to change the protocol to HTTP:// or HTTPS:// and the gateway will transition from a Configuring state to a Configured state.
  • Omitting URI when defining a component may result in unexpected behavior (7308)

    When defining components, failure to define the protocol and URI fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

    If you don’t include a URI for one component, and then for a subsequent component you define a URI without designating the protocol, the currentStatus.state.conditions field in the JSON response may not be present for the second component, for which the current state is isError. This makes it challenging to determine the root cause of the issue.

  • REGEX can only be used for path-based URIs and only in components (5747)

    Gateways or components that use match methods of REGEX or REGEX_CASE_SENSITIVE are failing either by erroring out or by getting stuck in a configuring state.

    Ingress URIs using a match method of REGEX or REGEX_CASE_SENSITIVE can be used only in a component (and not in gateways), and will work only when the URIs specify relative paths starting with / or \/.

    For example:

      "uris": {
        "/*.jpg": {
          "matchMethod": "REGEX"
        }
      },
    

    Workaround:

    • If you have a gateway that is unresponsive for more than a few minutes, make sure it doesn’t use REGEX or REGEX_CASE_SENSITIVE as a match method in any of its URIs.
    • If you have a component that is unresponsive, make sure that it complies with the regex guidelines above if REGEX or REGEX_CASE_SENSITIVE is used as a match method.
    • If you need to change the match method to resolve regex issues, you can use the NGINX web interface or send a PUT request using the API to the appropriate gateway or component, and specify a match method of EXACT, PREFIX, or SUFFIX. For a gateway, change your URI string to a string that specifies a full protocol–HTTP or HTTPS–and host. For components, the URI string can specify a relative path; a full protocol and host; or a full protocol and host and path.
  • Web user interface incorrectly shows default error log state as enabled (7447)

    The default setting for the errorLog for a component is Disabled; however, the web interface shows the setting as Enabled when no action has been taken to change it.

    Workaround:

    • Explicitly set the errorLog to the state that is desired, eitherDisabled or Enabled. The web interface will then display an errorLog state that is consistent with what is deployed on the component.
  • Current status does not always give an accurate account of the NGINX Controller configuration (7497)

    Current status does not always reflect the NGINX Controller configuration. This happens, for example, when multiple actions are scripted in a short period (less than 10 seconds) for the same NGINX instance. Examples of inaccurate representations:

    • Components showing that they are in the isConfigured state when the configuration is still being deployed
    • Components don’t show that they are in the isDeleting state while the configuration is being deleted

    Workaround:

    • Avoid scripting multiple actions that run in combination in a short period.
  • Intermediate certificates not pushed to NGINX Plus instance (7500)

    When you upload a certificate chain that includes a CA cert, then associate that cert with a gateway or component, only the end-entity certificate (server cert) is pushed to the NGINX Plus instance. The intermediate CA certs are not pushed when present. As a result, the chain of trust cannot be established.

    Workaround:

    • Manually deploy certificates and keys to the NGINX Plus instance’s local filesystem. For example, copy the key file to NGINX Plus at /etc/ssl/mycert.key, and the server cert along with all cacerts in a single file at /etc/ssl/mycert.crt.

      On NGINX Controller, create a cert object of type REMOTE_FILE (use “Add a remote file” in the UI). Use the full path to the files on the NGINX Plus instance for privateKey and publicCert.

  • Gateway accepts a relative path instead of a URI (7212)

    When creating or updating a gateway, you must provide a URI as the value for the URI field; URI’s MUST start with a protocol scheme HTTP or HTTPS. If you provide a relative path you will not see any error messages, but NGINX Controller will not write the desired config to the NGINX Plus instance(s).

    Workaround:

    • Do not provide a relative path when defining the URI field for a gateway.

      When defining components, failure to have at least one URI and having URIs with missing protocol fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

  • Referencing a non-existing identity provider for a component does not generate an error (7397)

    Referencing a non-existing identity provider when defining a component does not generate an error. The status for the component will remain at Configuring.

    Workaround:

    • Remove the reference to the non-existing identity provider.
  • Can’t delete a configured gateway when there is a gateway in an error state (7058)

    When an entity– for example, a gateway, component, or published API–references an instance and is in an error state, requests to DELETE other entities that also reference the instance are accepted but not performed.

    Workaround:

    • Resolve any errors for the entity that references an instance, and then perform the DELETE request on the entity that you want to remove.

Infrastructure

  • Cannot delete an instance that doesn’t have NGINX Plus installed (7704)

    An instance added to NGINX Controller that doesn’t already have NGINX Plus installed cannot be deleted from NGINX Controller.

  • Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)

    The Instance Overview page in the NGINX Controller web interface can become unresponsive if the network interface for any data plane instance is configured only for IPv6. The Instance Overpage page hangs and continually displays the message “Loading Instances.”

  • Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)

    When a create, update, or delete request is sent to a gateway or component of an instance that cannot be reached, the currentStatus.state.selfConfigState reports true for isConfiguring (for create or update) and true for isDeleting (for delete) indefinitely.

    The state will advance if and only if the instance becomes reachable.

  • Controller Agent stops running on CentOS 6.x (1659)

    When running the Controller Agent on CentOS 6.x the Agent starts and then stops.

    Workaround:

    • Edit the file /etc/controller-agent/agent.conf.
    • Add the lines level = INFO and level = DEBUG to the end of the file. This will keep the Controller Agent stable.
  • NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)

    If the compute VM for an NGINX Plus instance changes, NGINX Controller registers the instance as a new instance instead of honoring the existing agent.conf.

Platform

  • Session duration is limited to 8 hours (7239)

    The current session lifetime is limited to a maximum of 8 hours. This limit cannot be configured for longer or shorter durations. Once the session lifetime limit has been exceeded, users must log in again to receive a new session token.

  • Cannot add tags to users with the NGINX Controller web interface (7165)

    In NGINX Controller 3.0.0, you cannot add tags to users when creating or updating users using the web interface, and any tags that are created for a user using the API are not displayed.

  • NGINX Controller returns 429 response to new requests when the queue of unprocessed requests is above threshold (7526)

    NGINX Controller will return a response of 429 “Too many requests” to new PUT, POST, or DELETE requests when its queue depth of unprocessed requests is above a threshold.

    Workaround:

    • Retry the request to which the 429 was received after allowing time for the queue to be processed.
  • All users require the built-in Guest role (6231)

    It’s not possible to create a user without a role. By default, all users have the built-in Guest role when they’re created if another role isn’t specified. The Guest role has read access to environments.

Documentation

  • The API Reference documentation omits the “Get a Location” endpoint (7747)

    In the API Reference documentation located at <FQDN/docs/api/ on NGINX Controller, the GET Location endpoint is omitted. The missing information is provided below.

    GET <FQDN>/api/v2/infrastructure/locations/{locationName}
    

    Get a Location:

    • Returns information about a specified Location resource.
    • PATH PARAMETER(S): locationName
      • required
      • string
      • The name of the Location that you want to view.
  • Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)

    In NGINX Controller v3.0.0, the “Manage Apps” document (<FQDN>/docs/services/apps/manage-apps/) contains a line that states “CRUD (Create, Read, Update, Delete) operations on an application are determined by RBAC role authorization.”

    This statement is incorrect. RBAC permissions are not applied at the Apps level in this release. Instead, the ability to perform CRUD operations is granted by the legacy Admin or User Roles.

  • Stack trace errors occur in the API reference docs (6409)

    In NGINX Controller v3.0.0, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/3.0.0.

    For several elements – most notably the login credentials schema – clicking on an object to expand it will result in a ReDoc stack trace error.

    Workaround:

    • Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.
  • Links to metrics descriptions from user interface point to the metrics reference landing page (7255)

    When you create a new Dashboard using the NGINX Controller user interface, you can click on the “i” (info) symbol to view a description of specific metrics. Clicking the info symbol should bring you to the metrics catalog entry for that metric. In v3.0.0 of NGINX Controller, all of the links point to the metrics reference landing page.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R19
  • NGINX Plus R20

NGINX Controller Version 2.9.0

These release notes provide general information and describe known issues for NGINX Controller version 2.9.0, in the following categories:

Updates

Analytics

  • None

Platform

  • Bug Fixes

Load Balancer

  • Bug Fixes

API Management

  • Preview: Developer Portal:

    This release includes a preview of the API Management Module’s Developer Portal feature. Admins can choose to add API documentation – including descriptions, sample requests, and sample responses – for APIs published through Controller. Users can view the API documentation in the preview of the dev portal.

Known Issues

  • Kubernetes components that remained configured/running after failed Controller installation prevented reinstall:

    When a Controller installation failed after the kubernetes components were installed, the Kubernetes processes continued to run and configuration was not reset. Subsequent attempts to install the Controller fail because required ports are already in use.

    Workaround:

    1. Check the output of the failed install process and the install log in /var/log/nginx-controller/ to determine why the install process failed. Often, this can occur due to database connectivity or authentication issues.

      You will need to resolve this root cause prior to attempting reinstallation of controller.

    2. When you are ready to reinstall, run the commands shown below and then follow the prompts to remove the Kubernetes configuration.

      sudo kubeadm reset
      mv $HOME/.kube $HOME/.kube-old
      
    3. Proceed with re-installing the Controller.

  • Controller 2.9 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.9 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

  • API Management module allows upload of multiple JWT authentication policies for the same environment:

    When two JWT client group authentication policies are applied to a single environment, publishing to the gateway will fail. Workaround:

    Remove the additional JWT authentication policy, then try to publish the config.

  • Upgrade from 2.6.0 Incorrectly Migrated TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 incorrectly migrated the API Management TLS/SSL Policy resources. Subsequent attempts to edit and publish the APIM config failed.

    Workaround:

    • Create a new TLS/SSL Policy and add it to the affected endpoint.
    • Remove the broken TLS/SSL Policy from the endpoint.
    • Save and publish the APIM config.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file.

    Workaround:

    If you have made manual edits to the agent.conf file, take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps of the New Instance dialog.
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf
    4. Restart the controller-agent process.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15
  • NGINX Plus R16
  • NGINX Plus R17
  • NGINX Plus R18
  • NGINX Plus R19

NGINX Controller Version 2.8.1

Resolved Issues

  • Upgrade from Controller v2.7.3 Removed JWT Policy from API Management Config

    The upgrade from Controller v2.7.3 to Controller v2.8.0 removed a JWT policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. This issue is resolved in Controller v2.8.1.

  • Unable to Publish API Management Config with TLS/SSL Policy

    In the Controller-2.8.0 release, attempts to publish an API Management configuration with a TLS/SSL policy failed with an error message indicating that the certificate could not be loaded. This issue is resolved Controller-2.8.1.

Known Issues

  • Upgrade from 2.6.0 Incorrectly Migrated TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 incorrectly migrated the API Management TLS/SSL policy resources. Subsequent attempts to edit and publish the APIM config failed.

    Workaround:

    • Create a new TLS/SSL policy and add it to the affected endpoint.
    • Remove the broken TLS/SSL policy from the endpoint.
    • Save and publish the APIM config.
  • API Management Config Publication Failed After Renaming a JWT Client Group:

    Renaming a client group breaks the link between the client group and its previously published JWT policy. Subsequent attempts to publish the API Management config failed.

    Workaround:

    Do not rename any client groups that are referenced in an active JWT policy.

    If you have already done so, take the following steps:

    1. Delete the policy that references the renamed client group.
    2. Delete the client group.
    3. Create a new client group.
    4. Associate the relevant .jwk file with the new client group.
    5. Configure a JWT policy that references the new client group.
    6. Publish the config.
  • Controller 2.8 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.8 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
    4. Restart the controller-agent process.

NGINX Controller Version 2.8.0

These release notes provide general information and describe known issues for NGINX Controller version 2.8.0, in the following categories:

Updates

  • NGINX Controller runs on Kubernetes:

    This release introduces Kubernetes as an orchestration tool for the NGINX Controller containers.

  • Support for Debian 8 is deprecated in this release.

Analytics

  • None

Platform

  • Bug Fixes

Load Balancer

  • Bug fixes

API Management

  • JWT Client Groups now available:

    API Management users can choose between using API keys and JWTs for Client Groups. JWT Client Groups allow users to specify a .jwk by providing a URL.

  • Client Group API keys can be stored as hashed values:

    API keys can now be stored as SHA-256 hashed values on each API gateway. The JavaScript module njs must be installed on each NGINX Plus instance to enable this feature.

  • Updated Default Log Format for API Management:

    The default log format for API Management enables the full range of metrics for graphs, dashboards, and alerts. This resolves a previous issue where publishing an APIM configuration overwrote custom log formats.

Known Issues

  • NGINX Controller 2.8 requires specific Kubernetes packages to be held at version 1.15.3:

    After installing the Controller, exclude these packages from package upgrades:

    • kubeadm
    • kubectl
    • kubelet On Centos or Red Hat Enterprise Linux:
    1. Install yum-plugin-versionlock:

      yum install yum-plugin-versionlock

    2. For each package to exclude, run yum versionlock <package name>.

    On Debian or Ubuntu:

    1. For each package to exclude, run apt-mark hold <package name>.
  • Swap must be disabled on Controller host prior to install/upgrade:

    Per the Kubernetes requirements, you must disable swap on the host where Kubernetes will run. This must be done prior to installing the Nginx Controller or upgrading from a previous version (2.7.0 or earlier) of the Controller.

    You can find more information about this requirement in the Kubernetes kubeadm documentation.

    Please consult your OS Administrator and/or OS-specific documentation for more information about how to permanently disable swap on your machine.

  • Direct Upgrade from Controller 2.0.0 to Controller 2.8.0 fails:

    If you are using v2.0.0 and follow the upgrade process to install v2.8.0, the process will fail with an error.

    Workaround:

    Upgrade to Controller v2.7.3 first. Then, upgrade to Controller v2.8.0.

  • The API Management module does not support IP addresses when setting the location of a remote .jwk file for JWT Client Groups:

    The API Management module supports the use of JWT for Client Groups beginning in this release. You can either upload a local .jwk file or specify the URL via which a remote file can be accessed. If you provide an IP address when specifying the remote URL, a validation error that states the input is invalid will occur. The remote key file can only be saved if the URL is specified as a fully-qualified domain name (FQDN) .

    Workaround:

    Provide the FQDN and path to the .jwk file instead of using the server IP address.

  • API Management module allows upload of multiple JWT authentication policies for the same environment:

    The API Management module supports the use of JWT for Client Groups beginning in this release. When creating a Client Group, the system will allow you to create more than one JWT authentication policy for a single environment. If two JWT client group authentication policies are applied to a single environment, publishing to the gateway will fail. When you try to publish the config, you will see the following error: “Only one JWT policy can be added to an environment”.

    Workaround:

    Remove the additional JWT authentication policy and then try to publish the config.

  • Upgrade from Controller v2.6.0 Removed TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 removed a TLS/SSL policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. Attempts to add a new TLS/SSL policy and then publish the config also failed.

    Workaround:

    Remove, then re-create, the environment that contains the affected policy.

  • Controller 2.8.0 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.8.0 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
    4. Restart the controller-agent process.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17, R18, R19*

- NGINX Controller 2.8 doesn’t provide GUI support for the status_zone directive in the location block for R19.


NGINX Controller Version 2.7.0

These release notes provide general information and describe known issues for NGINX Controller version 2.7.0, in the following categories:

Updates

Analytics

  • This release introduces the new NGINX Controller Analytics module. With an Analytics module license, users can monitor, troubleshoot, and get insights into their NGINX Plus environments. For more information, please reach out to [email protected].

Platform

  • Bug Fixes

Load Balancer

  • Bug fixes

API Management

  • Read-Only Mode for API Management:

    User accounts with the “Read-Only” role are able to view API Management configurations but cannot make any edits.

  • Default Proxy Mode:

    HTTP keepalive policies can now be applied to upstream groups. Proxied requests are now made with HTTP/1.1 (previously HTTP/1.0).

  • Audit Log:

    The Audit Log lets you view all actions performed by Controller users over a set date range. Audit log results can be sorted and filtered by each column.

  • Reusable TLS Policies:

    TLS policies are now managed independently and can be associated with one or more Entry Points. A TLS policy includes the protocols, cipher settings, and session cache settings. The TLS certificate and private key are still Entry Point attributes.

Known Issues

  • Upgrade from Controller v2.6.0 Removed TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 removed a TLS/SSL policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. Attempts to add a new TLS/SSL policy and then publish the config also failed.

    Workaround:

    Remove, then re-create, the environment that contains the affected policy.

  • Controller version displayed in UI is v2.7.3:

    For the 2.7.0 release, the Controller UI shows the version number as 2.7.3.

  • Controller 2.7 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.7 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Upgrade the Agent to the latest version by following the instructions provided below to bring the system back into working order.

  • Audit Log’s “Apply” button is disabled when redefining a custom range:

    Steps to reproduce:

    • Create a custom date range on the new Audit Log page and click “Apply”.
    • Make changes to the custom date range.
    • The “Apply” button is not active.

    Workaround:

    • Select and apply a predefined date range, then select a custom range.
  • Upgrade to 2.7.0 doesn’t remove unused Docker container:

    During an update from any earlier version of the controller to v2.7, a previously-used Docker container is left running and emits a warning during the upgrade as follows:

    “Found orphan containers (controller_amplify-extapi_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the –remove-orphans flag to clean it up.”

    In the next release, this container will be cleaned up automatically. To remove it manually in v2.7, you can run the commands shown below:

    sudo docker stop controller_amplify-extapi_1
    sudo docker rm controller_amplify-extapi_1
    
  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
    4. Restart the controller-agent process.

  • Documentation for upgrading the controller agent is out-of-date:

    The instructions provided in the Admin guide and in the in-product documentation contain outdated instructions for upgrading the controller agent.

    To upgrade the agent, you should run the commands provided in the “Add new instance” dialog from your agent host. This will download and run a newer version of the agent.

  • Delay in API initialization post upgrade:

    For a period of about five minutes following updating the controller installation, a generic “Unknown” error banner appears in the UI when accessing the API Management pages. Approximately five minutes following update completion, this self-resolves and the application behaves normally.

    Workaround: You can manually dismiss the “Unknown” error banner.

  • Deletion of API Definition not reflected in GUI until page refresh:

    When you delete an API Definition in the API Module (which is present only if licensed), the deleted definition still appears in the GUI. The definition is, in fact, being deleted correctly. The UI is not updating to reflect this fact.

    Workaround: You can refresh the page, or navigate away and then return, to view the correct state of the application.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17, R18

NGINX Controller Version 2.6.0

These release notes provide general information and describe known issues for NGINX Controller version 2.6.0, in the following categories:

Updates

Platform

  • Bug fixes

Load Balancer

  • Bug fixes

API Management

  • Global DNS Resolver: Configure an external resolver that can be optionally enabled within each Upstream Group.

Known Issues

  • API Management policies applied may inherit a broader scope than intended.

    To resolve this issue, upgrade to Controller version 2.7.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
    4. Restart the controller-agent process.
  • Documentation for upgrading the controller agent is out-of-date:

    The instructions provided in the Admin guide and in the in-product documentation contain outdated instructions for upgrading the controller agent.

    To upgrade the agent, you should run the commands provided in the “Add new instance” dialog from your agent host. This will download and run a newer version of the agent.

  • API Management returns 504 Gateway Timeout for a period of minutes following upgrade from 2.5.0 to 2.6.0:

    For a period of about five minutes following updating the controller installation, a generic “Unknown” error appear in the GUI when accessing the API Management pages. Approximately five minutes following update completion, this self-resolves and the application behaves normally.

    Note that the “Unknown” error, which presents as a banner in the UI, must be manually dismissed.

  • Deleted API definition remains on the page:

    When you delete an API Definition in the API Module (present only if licensed), the deleted definition still appears in the GUI. The definition is, in fact, being deleted correctly. The UI is not updating to reflect this fact. You can refresh the page or navigate away and then return to show the correct state of the application.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17, R18

NGINX Controller Version 2.5.0

These release notes provide general information and describe known issues for NGINX Controller version 2.5.0, in the following categories:

Updates

Platform

  • New end user license agreement

Load Balancer

  • Bug fixes

API Management

  • Notifications will appear as you approach and exceed your license limit
  • To view license status , click on your API Management license within Settings > Licenses

Known Issues

  • Clients deploying controller-agent on CentOS 6 need to manually modify the logging level of controller-agent to ERROR level from INFO and restart controller-agent.
  • Administrators who delete a JWT policy will still see that policy in the environment select list. They should not select/configure a deleted policy.
  • NGINX Plus instances managed by the API Management module must be associated with a single Entry Point
  • Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17, R18

NGINX Controller Version 2.4.0

These release notes provide general information and describe known issues for NGINX Controller version 2.4.0, in the following categories:

Updates

Platform

  • Support contact information is now accessible from User Menu > Support

Load Balancer

  • The Instances page is now read-only, displaying the configuration that is currently running on each instance. This will help ensure nobody accidentally overwrites expected policy. To edit instance configurations, you must use Configuration Templates (Load Balancing > Configurations).
    • If an instance is associated to a Load Balancing Configuration Template or an API Management Entry Point, that information is reflected at the bottom of the page.
    • Instances that are not associated with a module can be promoted to Configuration Templates at the bottom of the Instances page for load balancing purposes, or associated with an Entry Point for API Management purposes.
  • Configuration Templates > Instances
    • You can now associate instances to a Configuration using tags.
    • Last Modified timestamp now shows the last time the configuration was changed, whether manually or through Controller.

API Management

  • Invalid Resource (404) Policy
    • From the Entry Point configuration page, you can specify an alternate HTTP error code, rather than the default 404 response to requests for invalid resources.

Known Issues

  • Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.
  • NGINX Plus instances managed by the API Management module must be associated with a single Entry Point.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17, R18

NGINX Controller Version 2.3.0

These release notes provide general information and describe known issues for NGINX Controller version 2.3.0, in the following categories:

Updates

Platform

  • Configure dashboard graph sets using one or more tags
  • Support spaces in upstream group name, route alias, system alias

Load Balancer

  • Configuration Templates
    • Ability to preview config file
    • Additional feedback around in-progress config push

API Management

  • Ability to edit environment policies
  • Usability improvements to API Definitions page
    • Entry point hostname color-coding:
      • Grey - config not pushed to the entry point
      • Green - config pushed, all associated instances are online
      • Yellow - config pushed, one or more instances offline & one or more instances online
      • Red - config pushed, all instances offline
    • Number of unrouted resources is displayed for each environment

Known Issues

  • Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17, R18

NGINX Controller Version 2.2.0

These release notes provide general information and describe known issues for NGINX Controller version 2.2.0, in the following categories:

Updates

Platform

  • Ability to configure the ServiceNow alert integration within Controller

Load Balancer

  • Configuration Templates - ability to name versions
  • Instances are sorted by display name (system alias, or hostname if no alias is set) in the inventory list and dropdown menus
    • Inventory menu in the app header defaults to sorting by display name, with options to change sort parameter

API Management

  • New “card” layout for the API Definition list view

Known Issues

No known major issues in this release

Supported NGINX Plus Versions

NGINX Controller works with the following NGINX Plus versions:

  • NGINX Plus R15, R16, R17

NGINX Controller Version 2.1.0

These release notes provide general information and describe known issues for NGINX Controller version 2.1.0, in the following categories:

Updates

Platform

  • Ubuntu 18 is qualified for hosting Controller Agents

Load Balancer

  • Configuration Templates can now be deleted

Known Issues

No known major issues in this release


NGINX Controller Version 2.0.0

These release notes provide general information and describe known issues for NGINX Controller version 2.0.0, in the following categories:

Updates

Load Balancer

  • Configuration Templates
    • Config
    • Associated Instances
    • Version History
    • Version Differencing
    • Arbitrary Reverting
  • Multi-push from ‘Load Balancing > Instances’
  • Notifications for multi-push status
  • ServiceNow alerting integration
  • Ability to enable error_log and access_log in virtual server
  • Renamed Amplify metrics
  • Rename tab from “Configure” to “Load Balancing”
  • Ability to tag instances from the ‘Load Balancing > Instances’ tab
  • Config-push deferral - ability to change a config and save it without pushing
  • Promote session persistence to its own section (Upstream Group config)
  • Offline installer / updater
  • Various bug-fixes

API Management

  • New API Management tab in Controller primary navigation
  • Create and manage API Definitions
  • Create and manage Upstream Groups
  • Create and manage Entry Points
  • Manage Clients and Client Groups
  • Policies can be applied to various objects in API Management

Known Issues

  • If upgrading from a previous version of Controller, password needs to be updated/reset.
  • Load Balancing - Instances page hangs when there are no hosts registered in Controller.
  • Load Balancing - Orange indicators will appear when you have unsaved changes, and may take several minutes to accurately reflect the correct status.
  • API Management - When configuring multiple rate limit policies for an Environment, each must use the same status code.

NGINX Controller Version 1.0.1

These release notes provide general information and describe known issues for NGINX Controller version 1.0.1, in the following categories:

Updates

  • Alerts - fix issue where editing an alert with threshold could result in an error
  • Analyzer - fix issue where old SSL cert could still be evident in the report alongside the new cert that replaced it
  • Analyzer - show appropriate message when port is missing for the listen directive
  • Configure - provide better error descriptions in the UI
  • Configure - fix issue where error notifications disappeared too quickly before they could be fully read
  • Configure - fix issue where user would see “undefined” error when loading a config that included a stream block
  • Configure - fix issue where virtual server SSL block is inactive after a page reload
  • Configure - fix issue where empty directives were not copied to the generated config file
  • Configure - fix issue where proxy is not allowed in regex location
  • Configure - fix issue where the validation for healthcheck URI does not allow for query params
  • Configure - fix issue of no validation for duplicate upstreams
  • Configure - fix the wrong visualization of healthcheck config after a push config action fails
  • Configure - fix error about server name being too long in the generated config

Known Issues

No known major issues in this release