NGINX Controller Releases

Analytics

• The NGINX HTTP Errors graph per instance renders 4xx and 5xx error stats for upstreams in the same color (16884)

  On the Infrastructure > Instances > Graphs page in the NGINX Controller web interface, the NGINX HTTP Errors graph per instance uses the same color for both 4xx and 5xx error statistics.

  Workaround:

  To see the 4XX errors, you can disable the 5XX errors by selecting http.status.5XX on the chart’s legend. Similarly, if you want to see the 5XX errors, disable the 4XX errors by selecting http.status.4XX on the chart’s legend.

Infrastructure

• New instances stay in Configuring state if they exceed the license limit (16881)

  When adding a data-path instance – for example, adding an NGINX Plus instance with the Controller Agent – that exceeds your license limit, the instance stays in a Configuring state and cannot be deleted. This unconfigured instance does not affect the ability to manage registered, licensed instances.

  Workaround:

  Contact NGINX Support via the MyF5 Customer Portal if you need to acquire more licenses. Once you have enough licenses, reinstall the Controller Agent on the instance that’s stuck in a Configuring state, using a different name for the instance than you used before. After the Controller Agent installs, the instance will register with NGINX Controller, where you’ll see the registered instance as well as the instance that’s stuck in a Configuring state. You can ignore the stuck instance.

Documentation

• The NGINX Controller OpenAPI Spec fails OAS 3 validation (16366)

  The NGINX Controller OpenAPI specification that’s provided as a download from in the API Reference guide fails OAS 3 validation and cannot be imported into NGINX Controller.

• Users endpoints aren’t shown in the NGINX Controller API reference guide (16705)

  When viewing the NGINX Controller API reference guide, the Users endpoints might not display in the endpoints list. When scrolling to the bottom of the list, the list ends at Services.

  Workaround:

  Expand the Services endpoints. The endpoints list will refresh, and the Users endpoints are shown just below Services.

• Add an Instance Using a Template procedure is missing a step (16920)

  The Add an Instance Using a Template procedure is missing a step. The complete procedure is as follows:

  1. Select the NGINX Controller menu icon, then select Infrastructure.
  2. On the Infrastructure menu, select Instances.
  3. On the Instances overview page, select Create.
  4. Select Create a new instance using a template.
  5. Add a name.
  6. Select a Location in the list, or select Create New to create a Location.
  7. Select an Instance Template in the list, or select Create New to create an Instance Template.
  8. Select Submit.

Installation and Upgrade

• NGINX Controller install script provides wrong commands for installing Docker CE on RHEL 7 (15666)

  When installing NGINX Controller on RHEL 7, the install script displays commands for installing Docker CE that are for are an older, unsupported version of Docker CE.

  Workaround:

  Before installing NGINX Controller on RHEL 7, you should install Docker CE 18.09 (recommended version).

• Custom Dashboards must be re-created after upgrading from NGINX Controller 3.7 to 3.8 (15815)

  When upgrading from NGINX Controller 3.7 to 3.8, Custom Dashboards are not migrated. You must re-create any Custom Dashboards after completing the upgrade.

Analytics

• Alerts set using NGINX Controller 3.6 or earlier need to be updated after upgrading to NGINX Controller 3.8 (14565)

  Alerts that are set using NGINX Controller 3.6 and earlier are not checked after upgrading to NGINX Controller 3.8; the “last checked” dates for these alerts are from before the upgrade.

  Workaround:

  To resume checking the alerts, go to the Analytics > Alerts page in the NGINX Controller user interface, then mute and unmute the alerts.

Apps and Services

• Create App Component settings for URI rewrites and header modification display URIs for non-referenced gateways (16103)

  The Applicable URIs list for URI rewrites and header modifications on the Create App Component page shows both referenced and non-referenced gateway URIs to choose from. If the URI for a non-referenced gateway is selected, NGINX Controller returns an error like the following: “Applied URI: <non-referenced URI> is not a part of the ingress URI.”

  Workaround:

  When selecting a gateway URI from the Applicable URIs list, select only URIs for gateways that the component references.

Infrastructure

• The Analyzer page reports 0 certificates when a cert is configured (14017)

  The Analyzer misevaluates SSL certificates associated with instances. The Analyzer page reports the number of certificates as 0 in the SSL block, even when certificates have been configured.

  Workaround:

  To see if your configuration is using a certificate:

  1. Open the NGINX Controller web interface and log in.
  2. From the NGINX Controller menu, select Services > Certs > Overview.
  3. On the Certs Overview page, select a certificate from the list. This opens a side panel where you can view the certificate’s details, including whether the certificate is associated with an app component and/or gateway.

• Once registered with NGINX Controller, a datapath instance cannot be renamed (15835)

  The name of your data-path instance – that is, the location where you install the NGINX Controller Agent – is fixed and cannot be changed once the instance is registered with NGINX Controller.

  When upgrading the Controller Agent, the install script allows entering a different instance name; however, this name is meant for adding new instances only. Attempting to provide another instance name when upgrading the Controller Agent could cause metrics to be misreported.

  Workaround:

  When upgrading the Controller Agent you can omit the instance name, or if you include the name, you must use the existing instance name.

• After upgrading to NGINX Controller 3.8, some statistics aren’t updated on the Graphs page (16064)

  After upgrading from NGINX Controller 3.7 to 3.8, you may notice that some statistics on the Instances Graphs page – in the web interface, select Instances > Graphs – aren’t updated for some configurations.

  Workaround:

  To resolve this issue, restart the nginx and controller-agent services on the NGINX Plus instance:

  • service nginx restart
  • service controller-agent restart

  After restarting these services, NGINX Controller should start reporting the statistics again.

Platform

• Per-environment and guest user access is broken for upgrades from NGINX Controller 3.4 or earlier (15941)

  Due to a change to the permission structure in NGINX Controller 3.4, upgrades from pre-3.4 versions to post-3.4 versions – including upgrades to NGINX Controller 3.8 – can break access for guest users and for users with “per-environment” roles.

  Workaround:

  There are two workarounds for this issue, depending on your upgrade history and risk tolerance.

  • Option 1 (recommended): If you have not yet upgraded NGINX Controller from a pre-3.4 release, first upgrade to NGINX Controller 3.4, and then upgrade again to the latest release.

  • Option 2: If you have already upgraded from a pre-3.4 release to a post-3.4 release without upgrading to 3.4 in the interim, you can only work around this issue by elevating the permissions of your guest users to regular users.

    Warning: This change involves a significant elevation of privilege and should be undertaken only with careful consideration. The regular user role has all the same admin role permissions except for the ability to promote new admins. If this change is acceptable for your installation – for any guest role users and any per-environment users with the guest role – these users should be edited by the default admin account to remove the guest role and replace it with the user role.

Documentation

• NGINX Controller Agent troubleshooting guide incorrectly refers to stub_status module (16028)

  In the NGINX Controller troubleshooting guide, which you can find at https://<Controller-FQDN>/docs/support/troubleshooting-controller, some of the information in the section covering metrics is incorrect.

  See the following steps and corrections:

  • Step 4. Make sure the Controller Agent is using the same user ID as the NGINX worker processes. The default user ID is nginx.

    Correction: The default user ID for the NGINX Controller Agent is root.

  • Step 8. Make sure stub_status is configured, and the stub_status module is included in the NGINX build. To check, run the following command:

    nginx -V
    

    The output should mention --with-http_stub_status_module.

    Correction: The NGINX Controller Agent gathers metrics via the NGINX Plus API, not via the stub_status module.

  Workaround:

  When troubleshooting issues with metrics, instead of checking that stub_status is enabled, you should verify that nginx api is enabled.

• The applies-to version list at the bottom of some docs doesn’t include NGINX Controller 3.8 (16795)

  At the bottom of the NGINX Controller docs, there’s a list of NGINX Controller versions that each doc applies to. For some docs installed with NGINX Controller 3.8, this list wasn’t updated. All of the docs installed with NGINX Controller 3.8 apply to that version.

API Management

• Code sample in the Dev Portal may have unintended URL encodings (13908)

  In some cases, the example code snippet in the Dev Portal is URL encoded when the API has path parameters. For example, the code snippet might look similar to the following:

  curl --request GET --url http://sportsapp.example.com/api/f1/drivers/%7BdriverId%7D
  

  Workaround:

  You must remove the URL encoding before running the example code snippet, like so:

  curl --request GET --url http://sportsapp.example.com/api/f1/drivers/{driverId}
  

• Known issues creating a Dev Portal using the NGINX Controller API (14385)

  • When passing a partial or incomplete theme customization payload to Dev Portal CRUD operations, the Dev Portal uses the missing settings’ default values. To customize fully, include the complete payload.
  • The Dev Portal supports only a curated list of fonts. You can see the list of accepted fonts while configuring a Dev Portal theme using the NGINX Controller web interface. If an invalid font value is submitted using the API, there’s no validation on the font value, and the default font is used instead.

  Workaround:

  Use the NGINX Controller web interface to construct a valid payload for creating and customizing the Dev Portal.

• Error status of Published API component affects publishing to Dev Portal (14452)

  The Dev Portal can be created and associated with a Published API at the same time. However, if the Published API is in an error state when the Dev Portal is created, an error is issued that incorrectly says that creating the Dev Portal failed. Then if the Dev Portal is resubmitted, the association fails with an “object exists” error. Furthermore, if the Published API is in an error state, publishing documentation to the Dev Portal for that API also fails.

  Workaround:

  Check the component status for the Published API before creating and associating a Dev Portal for it.

• Selecting between recent API Definitions and Dev Portals doesn’t refresh the webpage (14602)

  On the APIs page, selecting between items in the Recent API Definitions and Dev Portals quick list doesn’t refresh the page. Whichever item is displayed first is persistent.

  Workaround:

  To view specific API Definitions, select APIs > API Definitions. Then from the API Definitions list page, choose the API Definition that you want to view. Likewise, to view specific Dev Portals, select APIs > Dev Portals. Then from the Dev Portal list page, choose the Dev Portal that you want to view.

Infrastructure

• Configuration changes are not deployed to all instances when one instance is unavailable (14227)

  In multi-instance environments, when a configuration change cannot be deployed to one instance, such as when an instance is not available, the configuration is not deployed to any of the other instances.

• Controller Agent fails to download when deploying an instance to AWS on RHEL 7 (14886)

  When deploying an instance to Amazon Web Services (AWS) on RHEL 7, the installation fails, and the following error is logged to /var/log/agent_install.log: “Failed to download the install script for the agent.”

  Workaround:

  • Ensure that ports 443 and 8443 are open between NGINX Controller and the network where the instance is being deployed.
  • Verify that NGINX Controller can be accessed from the instance using the FQDN that was provided when NGINX Controller was installed
  • Ensure that the image referenced in the instance_template has a cURL version of 7.32 or newer.

• Upgrading Controller Agent returns an error that named instance already exists (14932)

  When Upgrading the Controller Agent, the install script warns that the named instance already exists and says to choose another name or remove the existing instance.

  Workaround:

  This error can be ignored for Controller Agent upgrades.

Documentation

• Note for PUT Update An Authentication Provider is incorrect (14576)

  In the NGINX Controller API reference guide, the description for the PUT /platform/auth/providers/{providerName} endpoint incorrectly says, “You must provide the full request body, including all configuration options, in a PUT request.”

  As long as the required parameters are included in the PUT request, the update works.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for each issue.

• Command to install Controller Agent includes --no-check-certificate by default (4166)
  • Solution Article: K31150658 | CVE-2020-5909
• NATS services do not require authentication (6404)
  • Solution Article: K59209532 | CVE-2020-5910
• On Debian and Ubuntu, Kubernetes packages are downloaded over plaintext HTTP (11771)
  • Solution Article: K84084843 | CVE-2020-5911

Additional Fixes

• REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)
• When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)
• Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)
• Component URI rewrites that are added and removed using the user interface cause a rewrite error (12527)
• The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)
• Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)
• API resources with POST method must be defined using the NGINX Controller API (12652)

Installation and Upgrade

• Controller Agent warns that Instance name already exists during upgrade (13350)

  When upgrading the Controller Agent, the install.sh script issues a warning that the Instance name already exists. This validation is a safeguard for new installations. You can safely disregard this warning for upgrades.

• Cannot log in to NGINX Controller when admin email address contains uppercase letters (13775)

  The installer for NGINX Controller allows users to create admin accounts with email addresses that contain uppercase letters. However, NGINX Controller expects emails to be only lowercase and prevents admin accounts with differently formatted email addresses from logging in.

  Workaround:

  You can reinstall NGINX Controller and enter a lowercase email address for the admin account. If that doesn’t work, contact NGINX Support for assistance.

API Management

• Published APIs do not go into an error state when the Instance they are using is deleted (10616)

  If you delete the last Instance serving a published API, and then edit the published API, the published API gets stuck in the Configuring state. Attempting to delete the published API causes the published API to get stuck in a Deleting state.

• Rate-limit policies are lost after upgrading to NGINX Controller v3.6 (12962)

  Existing rate-limiting policies applied to published APIs are lost when upgrading to NGINX Controller 3.6. The policies must be reconfigured at the Component level following the upgrade.

• JWK files need to be resubmitted after upgrading from NGINX Controller v3.4 to v3.6 (13289)

  Existing JWK files applied to Identity Providers are lost when upgrading from v3.4 to v3.6 and must be resubmitted following the upgrade.

• Importing API specs fails when done in the web interface (13501)

  Importing API specs from within the NGINX Controller user interface fails because of how the interface parses the text.

  Workaround:

  Import API specs using the NGINX Controller REST API.

Infrastructure

• Restarting the Controller Agent service leaves behind PID file with errors logged to agent.log (13390)

  When the Controller Agent service is restarted, the controller-agent.pid file is left behind, and errors about the PID file are logged in agent.log.

  Workaround:

  Manually remove /var/run/controller-agent/controller-agent.pid, then restart the Controller Agent service again. The Controller Agent will work normally after it restarts.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• The NGINX Controller web interface doesn’t have sufficient protection against a Cross-Site Request Forgery (CSRF) attack. (4882, 9721)
  • Solution Article: K31044532 | CVE-2020-5900
• The account recovery code for NGINX Controller that’s sent in email is not secure (11092)
  • Solution Article: K25434422 | CVE-2020-5899
• Undisclosed API endpoints may allow for a reflected Cross-Site Scripting (XSS) attack (12021)
  • Solution Article: K43520321 | CVE-2020-5901

Additional Fixes

• Cannot delete an instance that doesn’t have NGINX Plus installed (7704)
• NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)
• Cannot change SMTP password using the platform/global endpoint (11049)
• NGINX Controller v3 does not support RHEL 7 (11282)
• Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)
• Security section appears in the documentation but has no content (11294)

Installation and Upgrade

• Package error when installing nginx-plus-module-njs on CentOS 7 (12638)

  When installing nginx-plus-module-njs on CentOS 7, the installer returns a package error.

  Workaround:

  1. Uninstall the existing nginx-plus-module-metrics module:

     sudo yum remove -y nginx-plus-module-metrics
     

  2. Install the latest version of nginx-plus-module-njs:

     sudo yum install -y nginx-plus-module-njs
     

  3. Install the nginx-plus-module-metrics module, specifying the version as 1.19:

     sudo yum install -y nginx-plus-module-metrics-1.19*
     

  Verification:

  1. To confirm that the correct versions are installed, run the following command:

     yum list installed | egrep 'nginx|metrics'
     

     The expected output looks similar the following:

     avrd-libs.x86_64 0.15.5_1850598.release_3_5-1 @nginx-controller-metrics
     nginx-controller-agent.x86_64 3.5.0-10484161.el7 @nginx-controller
     nginx-plus.x86_64 22-1.el7.ngx @nginx-plus
     nginx-plus-module-metrics.x86_64 1.19.0-1850598.el7.ngx @nginx-controller-metrics
     nginx-plus-module-njs.x86_64 22+0.4.1-1.el7.ngx @nginx-plus
     

API Management

• The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)

  The URIs for a Published API use the PREFIX match method, regardless of the resource match method that was specified in the API Definition.

• API resources with POST method must be defined using the NGINX Controller API (12652)

  When defining an API Definition version with a resource that uses the POST method and specifies an example request, the generated API spec in the web interface is invalid.

  Workaround:

  Use the NGINX Controller API to define API Definition version resources that use the POST method and contain an example request in the API documentation.

Apps and Services

• Component URI rewrites that are added and removed using the web interface cause a rewrite error (12527)

  If a URI rewrite for a Component is added and then deleted using the NGINX Controller web interface, the API spec that’s sent to NGINX Controller includes an empty programmability section. This results in an Error status for the Component, and an error similar to the following is returned: "Error: Arguments of rewrite : incomingPattern or rewritePattern is missing."

  Workaround:

  • Option 1 (Recommended): Send a PUT request to the Components endpoint that excludes the programmability section.
  • Option 2: Configure a placeholder rewrite that writes from a pattern to a pattern. Using a pattern that doesn’t match any request is best.

Infrastructure

• Controller Agent installation doesn’t warn about duplicate instance names until an Instance has been registered with NGINX Controller (12370)

  The Controller Agent installation script typically fails with a warning if the provided instance name already exists for the specified location. However, a race condition occurs if the installation script is run on two instances at the same time with the same name for the same location. When this happens, the failed installation doesn’t issue a warning, but only one instance is added to NGINX Controller.

  Workaround:

  No workaround is necessary. The behavior is the same as if the installation script displayed the warning.

• Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)

  When the nginx-controller-agent package is removed on an NGINX Plus instance using apt-get-purge, the /etc/controller-agent directory and its contents are deleted. This directory is where NGINX Controller places certificates when NGINX Controller is used to configure and manage NGINX Plus instances. When this directory is deleted, the NGINX Plus instance becomes unresponsive and the NGINX service fails to start.

Platform

• Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)

  When a database certificate is added that uses an IP address instead of an FQDN, or if the certificate Common Name (CN) is invalid, the NGINX Controller database services reject the invalid cert and refuse database connections. In this case, the system becomes unresponsive until a valid cert is provided.

  Workaround:

  Use the hostname instead of an IP address, or use a certificate where the IP address is mentioned in Subject Alternative Name (SAN) as DNS type.

• NGINX Controller Agent and AVRD do not work on 32-bit distributions (12605)

  The Analytics, Visibility, and Reporting daemon (AVRD) is compiled as a 64-bit app and does not run on 32-bit distributions. Configuration Management using the Controller Agent is possible on 32-bit distributions; however, metrics are not reported.

  Workaround:

  Install the NGINX Controller Agent and AVRD on a 64-bit distribution.

  Important: NGINX Controller support for 32-bit distributions will be deprecated in a future release.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• Malformed messages lead to segmentation fault of the Analytics, Visibility, and Reporting daemon (AVRD) (9068)
  • Solution Article: K95120415 | CVE-2020-5895
• The Analytics, Visibility, and Reporting daemon (AVRD) is world-readable and writeable (9067)
  • Solution Article: K95120415 | CVE-2020-5895
• The NGINX Controller webserver does not invalidate the the server-side session token when users log out (8576)
  • Solution Article: K13028514 | CVE-2020-5894

Additional Fixes

• API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

Installation and Upgrade

• Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)

  When the conditions noted below are met, the Identity Provider API keys are not migrated and must be re-entered.

  Conditions:

  This issue applies when the following conditions are true:

  • You are running NGINX Controller v3.0.
  • You are running NGINX Controller v3.2, which you previously upgraded from NGINX Controller v3.0.
  • You are running NGINX Controller v3.3, which you previously upgraded from NGINX Controller v3.0.

  Workaround:

  Step 1: Copy and save your API Identity Profile keys

  Before upgrading, you must complete the following steps to copy and save any API Identity Profile keys that you have.

  1. Open the NGINX Controller web interface and log in.
  2. Select the NGINX Controller menu icon, then select Services.
  3. On the Services menu, select APIs.
  4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
  5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
  6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
  7. Copy the Identity Provider API key value and save it locally. You’ll need to re-enter this value after you upgrade.

  You can now complete the upgrade to NGINX Controller v3.4.

  Step 2: Restore your API Identity Profile keys

  After you’ve upgraded to NGINX Controller v3.4, you can restore the API Identity profile keys that you saved.

  1. Open the NGINX Controller web interface and log in.
  2. Select the NGINX Controller menu icon, then select Services.
  3. On the Services menu, select APIs.
  4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
  5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
  6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
  7. Paste the copied Identity Provider API key that you saved in the previous procedure into the key field. Then, click Save.

• NGINX Controller v3 does not support RHEL 7 (11282)

  NGINX Controller v3 does not support Red Hat Enterprise Linux (RHEL) v7. The NGINX Controller installation fails during the Docker version check. Manually updating the Docker version on RHEL 7 does not solve the problem.

Apps and Services

• Certificates imported out of order using drag-and-drop cause an error when creating a cert bundle (10761)

  When certificates are imported using the drag-and-drop feature in the NGINX Controller web interface, the system expects the first certificate that’s added to be the public certificate. Additional certificates that are added are treated as intermediate CA certs.

  If the certificates aren’t added in order, with the public certificate added first, the resulting API request has transposed values for the publicCert and caCert parameters.

  Workaround:

  When adding certificates using drag-and-drop, always drag the public cert file in first, followed by any additional CA certificates. The private key can be dragged in either before or after the certificates. Alternatively, you can choose to copy and paste the PEM text into the corresponding fields.

Platform

• Cannot change SMTP password using the platform/global endpoint (11049)

  Sending the SMTP password in a PATCH request to the platform/global endpoint has no effect.

  Workaround:

  To change the SMTP password, you can use the help.sh script that’s located in /opt/nginx-controller/ on the NGINX Controller host. For instructions, see the documentation that’s installed with NGINX Controller: https://<Controller-FQDN>/docs/platform/using-helper-script/#update-smtp-settings.

Documentation

• Security section appears in the documentation but has no content (11294)

  In NGINX Controller v3.4, a “Security” panel appears in the “Services” section of the onboard documentation. If you click on the Security panel, you will not see any content listed because there is no security documentation for this release. Users should disregard the “Security” panel in the documentation for this release.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• The NGINX Controller Agent installer script ‘install.sh’ uses HTTP instead of HTTPS to check and install packages (8587)
  • Solution Article: K00958787 | CVE-2020-5867
• The helper.sh script, which is used optionally in NGINX Controller to change settings, uses sensitive items as command-line arguments (7436)
  • Solution Article: K11922628 | CVE-2020-5866
• NGINX Controller uses insecure connection for communications with PostgreSQL database by default (6509)
  • Solution Article: K21009022 | CVE-2020-5865

Additional Fixes

• Creation of eventually consistent objects returns 201 instead of 202 (7052)
• Can’t delete a configured gateway when there is a gateway in an error state (7058)
• Dependencies Missing from Open Source Dependencies Report (7404)
• Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)
• Intermediate certificates not pushed to NGINX Plus instance (7500)
• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)
• Agent warns about failed connection to receiver after update (9152)
• Certificate expiration shows expired warning and expiration date that doesn’t match the final certificate (9167)
• Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)
• Login endpoint has inaccurate session information in the API Reference Guide (9482)
• A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

Apps and Services

• NGINX Controller web interface shows “Error fetching apps” after a restart (10013)

  In some cases, after NGINX Controller restarts, you might see an error in the web interface that says, “Error fetching apps: Network Error.”

  Workaround:

  Log out of the web interface and then log back in.

• Gateways configured with duplicate ingress URIs error out with condition of “config” (10425)

  When upserting a Gateway, if the ingress URI (desiredState.ingress.uris) is a duplicate of another Gateway’s ingress URI, the upsert errors out as expected; however, the condition type (state.conditions.type) that’s reported is simply config, instead of an explanation for the condition that’s occurred.

  "state": {
      "conditions": [
          {
              "type": "config" 
          }
      ],
      "selfConfigState": {
          "configured": 0,
          "configuring": 0,
          "deleting": 0,
          "error": 1,
          "isConfigured": false,
          "isConfiguring": false,
          "isDeleting": false,
          "isError": true,
          "total": 1
      }
  }
  

  Workaround:

  To resolve the error, remove the Gateway with the duplicate ingress URI that’s in the error state. You can do this by issuing a PUT request and applying a new config with a unique ingress URI, or you can issue a DELETE request to remove the Gateway with the duplicate ingress URI.

• Component fails to configure when its Gateway URI collides with another Gateway’s URI (10430)

  In situations where a Component for one Gateway uses the FQDN of a different Gateway on the same host, NGINX Controller doesn’t push the config to NGINX Plus.

  Scenario

  1. Create a Gateway for foo.com with the hostname http://foo.com, and deploy to NGINX Plus ‘Instance 2’.
  2. Create a Component for foo.com with the URI http://foo.com/, and associate the Component with the Gateway foo.com and assign a workload.
  3. Create a Gateway for bar.com with the hostname http://bar.com, and deploy to NGINX Plus ‘Instance 2’.
  4. Create a Component for bar.com with the URI http://foo.com/baz, and associate the Component with the Gateway bar.com and assign a workload.

  In this scenario, the configuration should expose the location for /baz. However, although the Component is reported as configured, nothing happens.

• App Services API Known Issues (10473)

  The App Services API has known issues for certain attributes and objects, in particular:

  • Socket parameters defined in the API should not be used at all. This includes the receivBufferSize and sendBufferSize parameters that are common to Gateway ingress and Component ingress.
  • Session persistence works only for SessionPersistenceCookie. Using SessionPersistenceRoute or SessionPersistenceCookieLearn results in configuration errors.
  • When configuring workload monitoring, and when using the API to check headers in the response, it’s necessary to check for an exact match for the header. Matching with a regular expression or checking only for the existence of a header does not work.
  • A Load Balancing method of HASH won’t work if it contains a variable such as $request_URI for the userKey.

  The existing URI Rewrite capability in the App Services API has the following limitations:

  • The Rewrite directive is not associated with a particular URI or location block in the generated configuration.
  • The break flag is not supported.

Analytics

• SELinux settings can prevent statistics flow from agents (10342)

  Depending on the SELinux settings, an NGINX instance might not have permission to submit statistics to your NGINX Controller.

  If you have a properly configured environment, application, and component for which no metrics are being passed to NGINX Controller after more than 30 seconds of traffic have transpired, try the suggested workaround steps.

  See the NGINX blog post Using NGINX and NGINX Plus with SELinux for additional general guidance.

  Workaround:

  1. On the instance(s) associated with your Gateway, run the following command as root:

     # grep nginx /var/log/audit/audit.log | audit2allow -M nginx`
     

  2. Then, run this command:

     # semodule -i nginx.pp
     

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• API allows for unauthenticated account creation (9428)
  • Solution Article: K14631834 | CVE-2020-5863
• NGINX Controller Agent uses insecure connection for communications between NGINX Controller and NGINX Plus instances by default (5899)
  • Solution Article: K27205552 | CVE-2020-5864

Additional Fixes

• Stack trace error occurs when accessing the Login credentials schema in the API reference docs. (6409)
• JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)
• Links to metrics descriptions from user interface point to the metrics reference landing page (7255)
• Usage of API key authentication policies requires the njs module on data-plane instances (7298)
• Omitting URI when defining a component may result in unexpected behavior (7308)
• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

Apps and Services

• Certificate expiration shows expired warning and expiration date that doesn’t match the end-entity certificate (9167)

  A certificate might appear as expired on the certificate detail page, although the Valid To date is in the future. This happens because the summary status and expiration date are calculated based on the entire certificate chain from which the certificate is derived. So, if you have an intermediate CA cert in your chain, the expiration summary status and summary expiration date may reflect the status/date of any intermediate certificate. If you see such a status or expiration date, you should check the status of your entire certificate chain, including that of any intermediate CA.

• A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

  A high volume of Create, Update, and Delete (CUD) operations – for example, more than 90 – performed on Gateways and Components could create a condition in which NGINX Controller becomes unresponsive.

  NGINX Controller limits queued operations at 100 and then returns an HTTP 429 error: “Too many requests; wait, and try again.” If, subsequently, additional CUD operations are submitted at a rapid rate (more than 1 per second), the queue may be overrun, creating a condition where no further changes can be processed without involvement from NGINX Support.

  Workaround:

  To avoid this situation, limit the amount of API Create, Update, and Delete (CUD) operations sent to the Component and Gateway endpoints to fewer than 100 open actions. The queue should clear 1 action per 7 seconds, on average.

Infrastructure

• Agent warns about failed connection to receiver after update (9152)

  After updating NGINX Controller and the Controller Agent to v3.2.0, the system warns that the Agent “Failed to connect to the receiver.” However, if you look in the agent.log file, the Agent is logging communication with the receiver, as expected.

  This warning occurs when a TLS certificate signed with a custom CA is used for the NGINX Controller. Before v3.2.0, the Controller Agent did not verify server certificates.

  To determine the status of the Agent, perform the following on the Agent host:

  • Check the log file /var/log/nginx-controller/agent.log for errors.

  • Check the status of the Controller Agent service:

    systemctl status controller-agent

    The output should include the status running.

• Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)

  If a TLS certificate for NGINX Controller is signed with a custom CA, the Agent fails to verify the cert and returns an error similar to the following:

  2020-03-09 20:38:28,803 [5089] MainThread failed POST "https://<fqdn>:8443/1.4/<uuid>/agent/",
  exception: "HTTPSConnectionPool(host='<fqdn>', port=8443): Max retries exceeded with url: /1.4/<uuid>/agent/
  (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))"
  

  Workaround:

  1. Place the CA on the Agent host.

  2. On the Agent host, edit the /etc/controller-agent/agent.conf file. In the cloud directive section, make the following changes:

     • Add or set the requests_ca_bundle option to point to the CA file.
     • Set verify_ssl_cert to True. (Note: If this setting is set to False, then server certificate validation is disabled, which we don’t recommend.)

     [cloud]
     ...
     verify_ssl_cert = True
     requests_ca_bundle = <path-to-ca-bundle-file>
     

  3. Save the changes to the agent.conf file.

  4. Restart the agent:

     systemctl restart controller-agent

  5. Verify that the Agent can connect to the NGINX Controller host:

     systemctl status controller-agent

     The output should include the status running. You can also check the log file /var/log/nginx-controller/agent.log for errors.

Documentation

• Login endpoint has inaccurate session information in the API Reference Guide (9482)

  The API reference documentation for the Login endpoint describes new session behavior that hasn’t been implemented yet. At this time, the default maximum session lifetime is 8 hours. Once the session lifetime limit is exceeded, you must log in to obtain a new session token.

Installation and Upgrade

• Could not initialize a Kubernetes cluster when installing NGINX Controller on CentOS 7 (5863)

  There is an issue in CentOS 7 that creates a mismatch between the optimum Docker settings and optimum Kubernetes settings. This mismatch prevents Kubernetes from initializing during the installation of NGINX Controller on CentOS 7.

  The NGINX Controller installation fails after Docker and Kubernetes are installed, with the following error: “execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster.”

  Workaround:

  • Solution one:
    • Run sudo yum update. This updates the components to the latest version and patches the issue.
  • Solution two:
    1. Run controller-installer/install.sh and follow the instructions. Allow the installation to fail. This establishes Docker and Kubernetes.
    2. Stop Docker: sudo systemctl stop docker
    3. Edit /etc/docker/daemon.json and make the the following change: native.cgroupdriver=systemd to native.cgroupdriver=cgroupfs
    4. Start Docker: sudo systemctl start docker
    5. Reset the Kubernetes configuration: sudo kubeadm reset -f
    6. Run controller-installer/install.sh and follow the instructions. Rerunning the installation does not update the version of any of the packages on the machine.

Apps and Services

• When committing a gateway with REGEX matchMethod, whose URI includes an explicit port and a path, the error message incorrectly states that the port is invalid (8452)

  In the context of configuring a gateway, only the protocol, domain, and optional port have any meaning. If the URI specified when configuring a REGEX matchMethod gateway includes a path, including a final / after the port, you will receive an error stating: Port value is invalid for hostname .*\\.example\\.com, where the hostname string corresponds to the data you provided for the domain.

  Workaround:

  • Specify your REGEX URI for a gateway as only the protocol, domain, and port with no trailing path.

• REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)

  The logic for interpreting REGEX and REGEX_CASE_SENSITIVE match methods for REGEX-based Component URIs is reversed.

  REGEX-based Component URIs are converted into a case-insensitive location block in nginx.conf when the URI is specified as a relative URI (leads with a path, not a domain) and when the match method is specified as REGEX_CASE_SENSITIVE. Conversely, when the match method is REGEX, the REGEX-based Component URI is interpreted as case sensitive.

  If you need your REGEX-based Component URI to be interpreted as case sensitive, then you can specify the match method as REGEX for now. However, keep in mind that when this logic mismatch is fixed in a later release of NGINX Controller, you’ll need to revert the match method accordingly.

• When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)

  In some cases, when communication between NGINX Controller and the Controller Agent is interrupted, an error similar to the following is returned: “Error: Failed to deploy to instance, Could not find the targeted NGINX running on the system None”. This error neglects to indicate the name of the system or systems for which the communication has failed.

  Workaround:

  • Communication difficulties can be persistent or temporary, depending on the nature of the failure. It is possible to see some more persistent communication failures by looking at the Infrastructure > Graphics pane in NGINX Controller. If the hostname and nginx-plus instance are down, these items will appear with a red background.

  • If all the systems and nginx-plus instances have with a white background, the communication problem could be temporary or intermittent, and won’t necessarily show in the interface.

    Things to check in order are:

    • Ensure the agent host/container is up
    • Ensure the controller-agent process is running
    • Ensure the nginx-plus process is running
    • Verify network communication between NGINX Controller and the Controller Agent on port 443.

Analytics

• Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)

  When upgrading from NGINX Controller 3.0.0 to 3.1.0, the app-centric metrics are not sent from the NGINX instances to NGINX Controller until a new configuration push is performed.

  Workaround:

  • Perform a Put to an object on each NGINX instance to update the config.

Documentation

• API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

  The API Management documentation states that the API Management module requires a dedicated NGINX Plus instance. This is no longer the case. As of v3.0, the API Management module no longer needs to run on a dedicated instance.

• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

  The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

  The following API actions are considered “experimental” features; use is not supported and functionality may change.

  • Create an Environment
  • Create an App
  • Create or update an App Component with the following properties defined in desiredState:
    • ingress.tcpKeepAlive
    • backend.workloadGroups.uris.failTimeout
    • backend.workloadGroups.uris.resolve
    • backend.workloadGroups.uris.route
    • backend.workloadGroups.uris.service
    • backend.workloadGroups.uris.slowStart
    • backend.workloadGroups.serviceDiscovery
    • backend.workloadGroups.serviceDiscovery.dnsResolver
    • backend.workloadGroups.serviceDiscovery.dnsResolver.uris
    • backend.preserveHostHeader
    • backend.queue
    • backend.httpVersion
    • backend.ntlmAuthentication
    • backend.persistentState
    • programmability.httpHttpsRedirect
    • programmability.uriRedirects
    • programmability.cookieModifications
    • programmability.requestHeaderModifications
    • programmability.responseHeaderModifications
    • security
    • security.waf
    • security.conditionalAuth
    • security.interceptWorkloadErrors
  • Create or update a Gateway with the following properties defined in desiredState:
    • ingress.headers
    • ingress.http2
    • ingress.spdy
    • ingress.proxyProtocol
    • ingress.setFib
    • ingress.fastOpen
    • ingress.backlog
    • ingress.acceptFilter
    • ingress.deferred
    • ingress.isIpv6Only
    • ingress.reusePort
    • ingress.notFoundStatusCode

• Stack trace errors occur in the API reference docs (8481)

  In NGINX Controller, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/.

  For a number of elements, clicking on an object to view the nested content will result in a ReDoc stack trace error.

  Workaround:

  Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.

• Dependencies Missing from Open Source Dependencies Report (7404)

  In NGINX Controller v3.1, the following open source dependencies were omitted from the open source acknowledgements PDF:

  • nats version 2.1.0 (https://github.com/nats-io/nats.c)
  • protobuf version 3.10.1 (https://github.com/protocolbuffers/protobuf)
  • protobuf-c version 1.3.2 (https://github.com/protobuf-c/protobuf-c)

API Management

• Data displayed by the user interface does not match that returned by the REST API for some API Management endpoints (7117)

  When working with API Definitions and Published APIs in the API Management module, the following mismatches between the NGINX Controller user interface and the REST API have been reported. These are dependent on where the user created or modified the affected objects.

  • Identity Provider - API Key: When you create an Identity Provider that uses API Key authentication by using the REST API, the name of the Identity Provider group will not display in the user interface. This behavior resolves if you modify the Identity Provider resource by using the REST API.
  • Identity Provider Client - API Key: When you create an Identity Provider Client that uses API Key authentication by using the user interface, you will receive a “404 - Not Found” error in response to an HTTP GET request for the resource.
  • Published API: When you create a Published API by using the user interface, the Environment, Application, Gateway, Policy, and Routes will be missing from the response to an HTTP GET request for the resource.
  • API Definition: When you create an API Definition by using the REST API, the name of the API Definition is not displayed in the user interface. When you create an API Definition by using the user interface, the URIs will be missing from the response to an HTTP GET request for the resource.

• Usage of API key authentication policies requires the njs module on data-plane instances (7298)

  In the NGINX Controller documentation for the API Management module, we state that “You must install the njs JavaScript module on each NGINX Plus instance to enable API key hashing” when using API key authentication for an Identity Provider. This information is not accurate.

  You must install the njs module on all NGINX Plus instances if you want to use API key authentication for any element of NGINX Controller (not just for API Management). This enables cryptographically-protected storage of API keys in the NGINX Plus configuration.

  If you do not install the njs module and use API key authentication, whether for API Management or elsewhere, the system may experience errors that are not reported in the user interface. See the NGINX Admin Guide for njs installation instructions.

• API definitions base path is not imported from OAS spec file (7671)

  After you use the API Management REST API to create an API Definitions from an OpenAPI 3.0 spec file, the base path is empty.

  Workaround:

  • You must manually edit the API Definition and provide the base path for API definitions that are created from imported OAS spec files.

• JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)

  Within a single Published API, configuring both an API key authentication policy and a JWT authentication policy results in an error, and you will not be able to publish.

  Workaround:

  • If you’re using the NGINX Controller user interface, only add one type of authentication policy (API Key or JWT) per Published API.
  • If you need to use an API key policy and a JWT policy within a single Published API, you can configure those policies using the external API, on condition that the keys are presented in the header or header/bearer, respectively.

Installation and Upgrade

• NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)

  When installing NGINX Controller, the install.sh script misidentifies the Docker version that’s installed as 19.03.5. NGINX Controller actually installs docker-ce 18.09 and docker-ce-cli 19.03.5. These versions are supported.

  Note: The Docker Command Line Interface (docker-ce-cli) is not an NGINX Controller requirement. Rather, it’s an interface for the administrator to use when working with the Docker Daemon (docker-ce).

Apps and Services

• Creation of eventually consistent objects returns 201 instead of 202 (7052)

  Component and gateway objects are considered eventually consistent and will not be immediately configured when created. Currently, these endpoints always return a 201 HTTP code (status created) after a PUT (create) or POST. A 202 HTTP code (status accepted) should be expected instead.

• User interface cannot be used to correct some error cases from direct API calls (7247)

  A direct API call can result in an errored state that cannot be subsequently corrected via the user interface. For example, when creating a component, if the protocol in the workload URI is missing, the component is created and goes into the isError state. When the user interface is then used to try to add the protocol, the Publish button never becomes green to allow the change to be pushed.

  Workaround:

  • Via the API, use a PUT request to correct the error.

• A component with no URIs will result in an errored deployment (7311)

  A component must have one or more URIs in its definition. If a component has no URIs, it will result in an error when being deployed, resulting in a failed deployment.

  Workaround:

  • Include one or more URIs in the definition of a component.

• Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)

  When component-level access logging is enabled, either through the web interface or via the public API, publishing the configuration gets stuck in an inConfiguring state. It never advances to a Configured state.

  This condition will block updates to the relevant gateway until the underlying condition with the component is resolved.

  Workaround:

  • Do not enable component-level access logging.

    If you find yourself in this situation, you can perform a PUT using the public API – or republish via the web interface – for the relevant component with component-level access logging disabled. This will result in a configured system and will allow other updates to proceed correctly.

• Component-based access logging format specification is non-functional (7646)

  When specified via the NGINX Controller public API (it is not available through the web interface), the format string for the component-level access log always errors out with an invalid string format error, regardless of the validity of the string that’s specified.

  Workaround:

  • For components to deploy and successfully be configured, avoid setting the component desiredState.logging.accessLog.format.

• Gateway ingress URIs must specify either HTTP:// or HTTPS://, otherwise the gateway becomes stuck in Configuring state (7416)

  Gateway ingress URIs must specify either HTTP:// or HTTPS:// as the protocol for the ingress.uri. If the protocol is mistyped in a POST or PUT for a gateway, the gateway can become stuck in a Configuring state that never resolves nor issues an error.

  Workaround:

  • If you mistyped the ingress URI protocol, you can submit another PUT request to change the protocol to HTTP:// or HTTPS:// and the gateway will transition from a Configuring state to a Configured state.

• Omitting URI when defining a component may result in unexpected behavior (7308)

  When defining components, failure to define the protocol and URI fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

  If you don’t include a URI for one component, and then for a subsequent component you define a URI without designating the protocol, the currentStatus.state.conditions field in the JSON response may not be present for the second component, for which the current state is isError. This makes it challenging to determine the root cause of the issue.

• REGEX can only be used for path-based URIs and only in components (5747)

  Gateways or components that use match methods of REGEX or REGEX_CASE_SENSITIVE are failing either by erroring out or by getting stuck in a configuring state.

  Ingress URIs using a match method of REGEX or REGEX_CASE_SENSITIVE can be used only in a component (and not in gateways), and will work only when the URIs specify relative paths starting with / or \/.

  For example:

    "uris": {
      "/*.jpg": {
        "matchMethod": "REGEX"
      }
    },
  

  Workaround:

  • If you have a gateway that is unresponsive for more than a few minutes, make sure it doesn’t use REGEX or REGEX_CASE_SENSITIVE as a match method in any of its URIs.
  • If you have a component that is unresponsive, make sure that it complies with the regex guidelines above if REGEX or REGEX_CASE_SENSITIVE is used as a match method.
  • If you need to change the match method to resolve regex issues, you can use the NGINX web interface or send a PUT request using the API to the appropriate gateway or component, and specify a match method of EXACT, PREFIX, or SUFFIX. For a gateway, change your URI string to a string that specifies a full protocol–HTTP or HTTPS–and host. For components, the URI string can specify a relative path; a full protocol and host; or a full protocol and host and path.

• Web user interface incorrectly shows default error log state as enabled (7447)

  The default setting for the errorLog for a component is Disabled; however, the web interface shows the setting as Enabled when no action has been taken to change it.

  Workaround:

  • Explicitly set the errorLog to the state that is desired, eitherDisabled or Enabled. The web interface will then display an errorLog state that is consistent with what is deployed on the component.

• Current status does not always give an accurate account of the NGINX Controller configuration (7497)

  Current status does not always reflect the NGINX Controller configuration. This happens, for example, when multiple actions are scripted in a short period (less than 10 seconds) for the same NGINX instance. Examples of inaccurate representations:

  • Components showing that they are in the isConfigured state when the configuration is still being deployed
  • Components don’t show that they are in the isDeleting state while the configuration is being deleted

  Workaround:

  • Avoid scripting multiple actions that run in combination in a short period.

• Intermediate certificates not pushed to NGINX Plus instance (7500)

  When you upload a certificate chain that includes a CA cert, then associate that cert with a gateway or component, only the end-entity certificate (server cert) is pushed to the NGINX Plus instance. The intermediate CA certs are not pushed when present. As a result, the chain of trust cannot be established.

  Workaround:

  • Manually deploy certificates and keys to the NGINX Plus instance’s local filesystem. For example, copy the key file to NGINX Plus at /etc/ssl/mycert.key, and the server cert along with all cacerts in a single file at /etc/ssl/mycert.crt.

    On NGINX Controller, create a cert object of type REMOTE_FILE (use “Add a remote file” in the UI). Use the full path to the files on the NGINX Plus instance for privateKey and publicCert.

• Gateway accepts a relative path instead of a URI (7212)

  When creating or updating a gateway, you must provide a URI as the value for the URI field; URI’s MUST start with a protocol scheme HTTP or HTTPS. If you provide a relative path you will not see any error messages, but NGINX Controller will not write the desired config to the NGINX Plus instance(s).

  Workaround:

  • Do not provide a relative path when defining the URI field for a gateway.

    When defining components, failure to have at least one URI and having URIs with missing protocol fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

• Referencing a non-existing identity provider for a component does not generate an error (7397)

  Referencing a non-existing identity provider when defining a component does not generate an error. The status for the component will remain at Configuring.

  Workaround:

  • Remove the reference to the non-existing identity provider.

• Can’t delete a configured gateway when there is a gateway in an error state (7058)

  When an entity– for example, a gateway, component, or published API–references an instance and is in an error state, requests to DELETE other entities that also reference the instance are accepted but not performed.

  Workaround:

  • Resolve any errors for the entity that references an instance, and then perform the DELETE request on the entity that you want to remove.

Infrastructure

• Cannot delete an instance that doesn’t have NGINX Plus installed (7704)

  An instance added to NGINX Controller that doesn’t already have NGINX Plus installed cannot be deleted from NGINX Controller.

• Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)

  The Instance Overview page in the NGINX Controller web interface can become unresponsive if the network interface for any data plane instance is configured only for IPv6. The Instance Overpage page hangs and continually displays the message “Loading Instances.”

• Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)

  When a create, update, or delete request is sent to a gateway or component of an instance that cannot be reached, the currentStatus.state.selfConfigState reports true for isConfiguring (for create or update) and true for isDeleting (for delete) indefinitely.

  The state will advance if and only if the instance becomes reachable.

• Controller Agent stops running on CentOS 6.x (1659)

  When running the Controller Agent on CentOS 6.x the Agent starts and then stops.

  Workaround:

  • Edit the file /etc/controller-agent/agent.conf.
  • Add the lines level = INFO and level = DEBUG to the end of the file. This will keep the Controller Agent stable.

• NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)

  If the compute VM for an NGINX Plus instance changes, NGINX Controller registers the instance as a new instance instead of honoring the existing agent.conf.

Platform

• Session duration is limited to 8 hours (7239)

  The current session lifetime is limited to a maximum of 8 hours. This limit cannot be configured for longer or shorter durations. Once the session lifetime limit has been exceeded, users must log in again to receive a new session token.

• Cannot add tags to users with the NGINX Controller web interface (7165)

  In NGINX Controller 3.0.0, you cannot add tags to users when creating or updating users using the web interface, and any tags that are created for a user using the API are not displayed.

• NGINX Controller returns 429 response to new requests when the queue of unprocessed requests is above threshold (7526)

  NGINX Controller will return a response of 429 “Too many requests” to new PUT, POST, or DELETE requests when its queue depth of unprocessed requests is above a threshold.

  Workaround:

  • Retry the request to which the 429 was received after allowing time for the queue to be processed.

• All users require the built-in Guest role (6231)

  It’s not possible to create a user without a role. By default, all users have the built-in Guest role when they’re created if another role isn’t specified. The Guest role has read access to environments.

Documentation

• The API Reference documentation omits the “Get a Location” endpoint (7747)

  In the API Reference documentation located at <FQDN/docs/api/ on NGINX Controller, the GET Location endpoint is omitted. The missing information is provided below.

  GET <FQDN>/api/v2/infrastructure/locations/{locationName}
  

  Get a Location:

  • Returns information about a specified Location resource.
  • PATH PARAMETER(S): locationName
    • required
    • string
    • The name of the Location that you want to view.

• Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)

  In NGINX Controller v3.0.0, the “Manage Apps” document (<FQDN>/docs/services/apps/manage-apps/) contains a line that states “CRUD (Create, Read, Update, Delete) operations on an application are determined by RBAC role authorization.”

  This statement is incorrect. RBAC permissions are not applied at the Apps level in this release. Instead, the ability to perform CRUD operations is granted by the legacy Admin or User Roles.

• Stack trace errors occur in the API reference docs (6409)

  In NGINX Controller v3.0.0, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/3.0.0.

  For several elements – most notably the login credentials schema – clicking on an object to expand it will result in a ReDoc stack trace error.

  Workaround:

  • Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.

• Links to metrics descriptions from user interface point to the metrics reference landing page (7255)

  When you create a new Dashboard using the NGINX Controller user interface, you can click on the “i” (info) symbol to view a description of specific metrics. Clicking the info symbol should bring you to the metrics catalog entry for that metric. In v3.0.0 of NGINX Controller, all of the links point to the metrics reference landing page.

Analytics

• None

Platform

• Bug Fixes

Load Balancer

• Bug Fixes

API Management

• Preview: Developer Portal:
  
  This release includes a preview of the API Management Module’s Developer Portal feature. Admins can choose to add API documentation – including descriptions, sample requests, and sample responses – for APIs published through Controller. Users can view the API documentation in the preview of the dev portal.

Analytics

• None

Platform

• Bug Fixes

Load Balancer

• Bug fixes

API Management

• JWT Client Groups now available:

  API Management users can choose between using API keys and JWTs for Client Groups. JWT Client Groups allow users to specify a .jwk by providing a URL.

• Client Group API keys can be stored as hashed values:

  API keys can now be stored as SHA-256 hashed values on each API gateway. The JavaScript module njs must be installed on each NGINX Plus instance to enable this feature.

• Updated Default Log Format for API Management:

  The default log format for API Management enables the full range of metrics for graphs, dashboards, and alerts. This resolves a previous issue where publishing an APIM configuration overwrote custom log formats.

Analytics

• This release introduces the new NGINX Controller Analytics module. With an Analytics module license, users can monitor, troubleshoot, and get insights into their NGINX Plus environments. For more information, please reach out to [email protected].

Platform

• Bug Fixes

Load Balancer

• Bug fixes

API Management

• Read-Only Mode for API Management:

  User accounts with the “Read-Only” role are able to view API Management configurations but cannot make any edits.

• Default Proxy Mode:

  HTTP keepalive policies can now be applied to upstream groups. Proxied requests are now made with HTTP/1.1 (previously HTTP/1.0).

• Audit Log:

  The Audit Log lets you view all actions performed by Controller users over a set date range. Audit log results can be sorted and filtered by each column.

• Reusable TLS Policies:

  TLS policies are now managed independently and can be associated with one or more Entry Points. A TLS policy includes the protocols, cipher settings, and session cache settings. The TLS certificate and private key are still Entry Point attributes.

Platform

• Bug fixes

Load Balancer

• Bug fixes

API Management

• Global DNS Resolver: Configure an external resolver that can be optionally enabled within each Upstream Group.

Platform

• New end user license agreement

Load Balancer

• Bug fixes

API Management

• Notifications will appear as you approach and exceed your license limit
• To view license status , click on your API Management license within Settings > Licenses

Platform

• Support contact information is now accessible from User Menu > Support

Load Balancer

• The Instances page is now read-only, displaying the configuration that is currently running on each instance. This will help ensure nobody accidentally overwrites expected policy. To edit instance configurations, you must use Configuration Templates (Load Balancing > Configurations).
  • If an instance is associated to a Load Balancing Configuration Template or an API Management Entry Point, that information is reflected at the bottom of the page.
  • Instances that are not associated with a module can be promoted to Configuration Templates at the bottom of the Instances page for load balancing purposes, or associated with an Entry Point for API Management purposes.
• Configuration Templates > Instances
  • You can now associate instances to a Configuration using tags.
  • Last Modified timestamp now shows the last time the configuration was changed, whether manually or through Controller.

API Management

• Invalid Resource (404) Policy
  • From the Entry Point configuration page, you can specify an alternate HTTP error code, rather than the default 404 response to requests for invalid resources.

Platform

• Configure dashboard graph sets using one or more tags
• Support spaces in upstream group name, route alias, system alias

Load Balancer

• Configuration Templates
  • Ability to preview config file
  • Additional feedback around in-progress config push

API Management

• Ability to edit environment policies
• Usability improvements to API Definitions page
  • Entry point hostname color-coding:
    • Grey - config not pushed to the entry point
    • Green - config pushed, all associated instances are online
    • Yellow - config pushed, one or more instances offline & one or more instances online
    • Red - config pushed, all instances offline
  • Number of unrouted resources is displayed for each environment

Platform

• Ability to configure the ServiceNow alert integration within Controller

Load Balancer

• Configuration Templates - ability to name versions
• Instances are sorted by display name (system alias, or hostname if no alias is set) in the inventory list and dropdown menus
  • Inventory menu in the app header defaults to sorting by display name, with options to change sort parameter

API Management

• New “card” layout for the API Definition list view

Platform

• Ubuntu 18 is qualified for hosting Controller Agents

Load Balancer

• Configuration Templates can now be deleted

Load Balancer

• Configuration Templates
  • Config
  • Associated Instances
  • Version History
  • Version Differencing
  • Arbitrary Reverting
• Multi-push from ‘Load Balancing > Instances’
• Notifications for multi-push status
• ServiceNow alerting integration
• Ability to enable error_log and access_log in virtual server
• Renamed Amplify metrics
• Rename tab from “Configure” to “Load Balancing”
• Ability to tag instances from the ‘Load Balancing > Instances’ tab
• Config-push deferral - ability to change a config and save it without pushing
• Promote session persistence to its own section (Upstream Group config)
• Offline installer / updater
• Various bug-fixes

API Management

• New API Management tab in Controller primary navigation
• Create and manage API Definitions
• Create and manage Upstream Groups
• Create and manage Entry Points
• Manage Clients and Client Groups
• Policies can be applied to various objects in API Management