NGINX Controller Releases

API Management

• Code sample in the Dev Portal may have unintended URL encodings (13908)

  In some cases, the example code snippet in the Dev Portal is URL encoded when the API has path parameters. For example, the code snippet might look similar to the following:

  curl --request GET --url http://sportsapp.example.com/api/f1/drivers/%7BdriverId%7D
  

  Workaround:

  You must remove the URL encoding before running the example code snippet, like so:

  curl --request GET --url http://sportsapp.example.com/api/f1/drivers/{driverId}
  

• Known issues creating a Dev Portal using the NGINX Controller API (14385)

  • When passing a partial or incomplete theme customization payload to Dev Portal CRUD operations, the Dev Portal uses the missing settings’ default values. To customize fully, include the complete payload.
  • The Dev Portal supports only a curated list of fonts. You can see the list of accepted fonts while configuring a Dev Portal theme using the NGINX Controller web interface. If an invalid font value is submitted using the API, there’s no validation on the font value, and the default font is used instead.

  Workaround:

  Use the NGINX Controller web interface to construct a valid payload for creating and customizing the Dev Portal.

• Error status of Published API component affects publishing to Dev Portal (14452)

  The Dev Portal can be created and associated with a Published API at the same time. However, if the Published API is in an error state when the Dev Portal is created, an error is issued that incorrectly says that creating the Dev Portal failed. Then if the Dev Portal is resubmitted, the association fails with an “object exists” error. Furthermore, if the Published API is in an error state, publishing documentation to the Dev Portal for that API also fails.

  Workaround:

  Check the component status for the Published API before creating and associating a Dev Portal for it.

• Selecting between recent API Definitions and Dev Portals doesn’t refresh the webpage (14602)

  On the APIs page, selecting between items in the Recent API Definitions and Dev Portals quick list doesn’t refresh the page. Whichever item is displayed first is persistent.

  Workaround:

  To view specific API Definitions, select APIs > API Definitions. Then from the API Definitions list page, choose the API Definition that you want to view. Likewise, to view specific Dev Portals, select APIs > Dev Portals. Then from the Dev Portal list page, choose the Dev Portal that you want to view.

Infrastructure

• Configuration changes are not deployed to all instances when one instance is unavailable (14227)

  In multi-instance environments, when a configuration change cannot be deployed to one instance, such as when an instance is not available, the configuration is not deployed to any of the other instances.

• Controller Agent fails to download when deploying an instance to AWS on RHEL 7 (14886)

  When deploying an instance to Amazon Web Services (AWS) on RHEL 7, the installation fails, and the following error is logged to /var/log/agent_install.log: “Failed to download the install script for the agent.”

  Workaround:

  • Ensure that ports 443 and 8443 are open between NGINX Controller and the network where the instance is being deployed.
  • Verify that NGINX Controller can be accessed from the instance using the FQDN that was provided when NGINX Controller was installed
  • Ensure that the image referenced in the instance_template has a cURL version of 7.32 or newer.

• Upgrading Controller Agent returns an error that named instance already exists (14932)

  When Upgrading the Controller Agent, the install script warns that the named instance already exists and says to choose another name or remove the existing instance.

  Workaround:

  This error can be ignored for Controller Agent upgrades.

Documentation

• Note for PUT Update An Authentication Provider is incorrect (14576)

  In the NGINX Controller API reference guide, the description for the PUT /platform/auth/providers/{providerName} endpoint incorrectly says, “You must provide the full request body, including all configuration options, in a PUT request.”

  As long as the required parameters are included in the PUT request, the update works.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for each issue.

• Command to install Controller Agent includes --no-check-certificate by default (4166)
  • Solution Article: K31150658 | CVE-2020-5909
• NATS services do not require authentication (6404)
  • Solution Article: K59209532 | CVE-2020-5910
• On Debian and Ubuntu, Kubernetes packages are downloaded over plaintext HTTP (11771)
  • Solution Article: K84084843 | CVE-2020-5911

Additional Fixes

• REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)
• When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)
• Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)
• Component URI rewrites that are added and removed using the user interface cause a rewrite error (12527)
• The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)
• Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)
• API resources with POST method must be defined using the NGINX Controller API (12652)

Installation and Upgrade

• Controller Agent warns that Instance name already exists during upgrade (13350)

  When upgrading the Controller Agent, the install.sh script issues a warning that the Instance name already exists. This validation is a safeguard for new installations. You can safely disregard this warning for upgrades.

• Cannot log in to NGINX Controller when admin email address contains uppercase letters (13775)

  The installer for NGINX Controller allows users to create admin accounts with email addresses that contain uppercase letters. However, NGINX Controller expects emails to be only lowercase and prevents admin accounts with differently formatted email addresses from logging in.

  Workaround:

  You can reinstall NGINX Controller and enter a lowercase email address for the admin account. If that doesn’t work, contact NGINX Support for assistance.

API Management

• Published APIs do not go into an error state when the Instance they are using is deleted (10616)

  If you delete the last Instance serving a published API, and then edit the published API, the published API gets stuck in the Configuring state. Attempting to delete the published API causes the published API to get stuck in a Deleting state.

• Rate-limit policies are lost after upgrading to NGINX Controller v3.6 (12962)

  Existing rate-limiting policies applied to published APIs are lost when upgrading to NGINX Controller 3.6. The policies must be reconfigured at the Component level following the upgrade.

• JWK files need to be resubmitted after upgrading from NGINX Controller v3.4 to v3.6 (13289)

  Existing JWK files applied to Identity Providers are lost when upgrading from v3.4 to v3.6 and must be resubmitted following the upgrade.

• Importing API specs fails when done in the web interface (13501)

  Importing API specs from within the NGINX Controller user interface fails because of how the interface parses the text.

  Workaround:

  Import API specs using the NGINX Controller REST API.

Infrastructure

• Restarting the Controller Agent service leaves behind PID file with errors logged to agent.log (13390)

  When the Controller Agent service is restarted, the controller-agent.pid file is left behind, and errors about the PID file are logged in agent.log.

  Workaround:

  Manually remove /var/run/controller-agent/controller-agent.pid, then restart the Controller Agent service again. The Controller Agent will work normally after it restarts.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• The NGINX Controller web interface doesn’t have sufficient protection against a Cross-Site Request Forgery (CSRF) attack. (4882, 9721)
  • Solution Article: K31044532 | CVE-2020-5900
• The account recovery code for NGINX Controller that’s sent in email is not secure (11092)
  • Solution Article: K25434422 | CVE-2020-5899
• Undisclosed API endpoints may allow for a reflected Cross-Site Scripting (XSS) attack (12021)
  • Solution Article: K43520321 | CVE-2020-5901

Additional Fixes

• Cannot delete an instance that doesn’t have NGINX Plus installed (7704)
• NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)
• Cannot change SMTP password using the platform/global endpoint (11049)
• NGINX Controller v3 does not support RHEL 7 (11282)
• Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)
• Security section appears in the documentation but has no content (11294)

Installation and Upgrade

• Package error when installing nginx-plus-module-njs on CentOS 7 (12638)

  When installing nginx-plus-module-njs on CentOS 7, the installer returns a package error.

  Workaround:

  1. Uninstall the existing nginx-plus-module-metrics module:

     sudo yum remove -y nginx-plus-module-metrics
     

  2. Install the latest version of nginx-plus-module-njs:

     sudo yum install -y nginx-plus-module-njs
     

  3. Install the nginx-plus-module-metrics module, specifying the version as 1.19:

     sudo yum install -y nginx-plus-module-metrics-1.19*
     

  Verification:

  1. To confirm that the correct versions are installed, run the following command:

     yum list installed | egrep 'nginx|metrics'
     

     The expected output looks similar the following:

     avrd-libs.x86_64 0.15.5_1850598.release_3_5-1 @nginx-controller-metrics
     nginx-controller-agent.x86_64 3.5.0-10484161.el7 @nginx-controller
     nginx-plus.x86_64 22-1.el7.ngx @nginx-plus
     nginx-plus-module-metrics.x86_64 1.19.0-1850598.el7.ngx @nginx-controller-metrics
     nginx-plus-module-njs.x86_64 22+0.4.1-1.el7.ngx @nginx-plus
     

API Management

• The match method for an API Definition resource is removed (defaults to PREFIX) when an API is published (12594)

  The URIs for a Published API use the PREFIX match method, regardless of the resource match method that was specified in the API Definition.

• API resources with POST method must be defined using the NGINX Controller API (12652)

  When defining an API Definition version with a resource that uses the POST method and specifies an example request, the generated API spec in the web interface is invalid.

  Workaround:

  Use the NGINX Controller API to define API Definition version resources that use the POST method and contain an example request in the API documentation.

Apps and Services

• Component URI rewrites that are added and removed using the web interface cause a rewrite error (12527)

  If a URI rewrite for a Component is added and then deleted using the NGINX Controller web interface, the API spec that’s sent to NGINX Controller includes an empty programmability section. This results in an Error status for the Component, and an error similar to the following is returned: "Error: Arguments of rewrite : incomingPattern or rewritePattern is missing."

  Workaround:

  • Option 1 (Recommended): Send a PUT request to the Components endpoint that excludes the programmability section.
  • Option 2: Configure a placeholder rewrite that writes from a pattern to a pattern. Using a pattern that doesn’t match any request is best.

Infrastructure

• Controller Agent installation doesn’t warn about duplicate instance names until an Instance has been registered with NGINX Controller (12370)

  The Controller Agent installation script typically fails with a warning if the provided instance name already exists for the specified location. However, a race condition occurs if the installation script is run on two instances at the same time with the same name for the same location. When this happens, the failed installation doesn’t issue a warning, but only one instance is added to NGINX Controller.

  Workaround:

  No workaround is necessary. The behavior is the same as if the installation script displayed the warning.

• Removing nginx-controller-agent package with apt-get purge causes NGINX Plus instance to become unresponsive (12603)

  When the nginx-controller-agent package is removed on an NGINX Plus instance using apt-get-purge, the /etc/controller-agent directory and its contents are deleted. This directory is where NGINX Controller places certificates when NGINX Controller is used to configure and manage NGINX Plus instances. When this directory is deleted, the NGINX Plus instance becomes unresponsive and the NGINX service fails to start.

Platform

• Database certs with an IP address or an invalid CN cause NGINX Controller to become unresponsive (12173)

  When a database certificate is added that uses an IP address instead of an FQDN, or if the certificate Common Name (CN) is invalid, the NGINX Controller database services reject the invalid cert and refuse database connections. In this case, the system becomes unresponsive until a valid cert is provided.

  Workaround:

  Use the hostname instead of an IP address, or use a certificate where the IP address is mentioned in Subject Alternative Name (SAN) as DNS type.

• NGINX Controller Agent and AVRD do not work on 32-bit distributions (12605)

  The Analytics, Visibility, and Reporting daemon (AVRD) is compiled as a 64-bit app and does not run on 32-bit distributions. Configuration Management using the Controller Agent is possible on 32-bit distributions; however, metrics are not reported.

  Workaround:

  Install the NGINX Controller Agent and AVRD on a 64-bit distribution.

  Important: NGINX Controller support for 32-bit distributions will be deprecated in a future release.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• Malformed messages lead to segmentation fault of the Analytics, Visibility, and Reporting daemon (AVRD) (9068)
  • Solution Article: K95120415 | CVE-2020-5895
• The Analytics, Visibility, and Reporting daemon (AVRD) is world-readable and writeable (9067)
  • Solution Article: K95120415 | CVE-2020-5895
• The NGINX Controller webserver does not invalidate the the server-side session token when users log out (8576)
  • Solution Article: K13028514 | CVE-2020-5894

Additional Fixes

• API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

Installation and Upgrade

• Identity Provider API keys are not migrated during full or incremental upgrades from NGINX Controller v3.0 to v3.4 (11286)

  When the conditions noted below are met, the Identity Provider API keys are not migrated and must be re-entered.

  Conditions:

  This issue applies when the following conditions are true:

  • You are running NGINX Controller v3.0.
  • You are running NGINX Controller v3.2, which you previously upgraded from NGINX Controller v3.0.
  • You are running NGINX Controller v3.3, which you previously upgraded from NGINX Controller v3.0.

  Workaround:

  Step 1: Copy and save your API Identity Profile keys

  Before upgrading, you must complete the following steps to copy and save any API Identity Profile keys that you have.

  1. Open the NGINX Controller web interface and log in.
  2. Select the NGINX Controller menu icon, then select Services.
  3. On the Services menu, select APIs.
  4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
  5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
  6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
  7. Copy the Identity Provider API key value and save it locally. You’ll need to re-enter this value after you upgrade.

  You can now complete the upgrade to NGINX Controller v3.4.

  Step 2: Restore your API Identity Profile keys

  After you’ve upgraded to NGINX Controller v3.4, you can restore the API Identity profile keys that you saved.

  1. Open the NGINX Controller web interface and log in.
  2. Select the NGINX Controller menu icon, then select Services.
  3. On the Services menu, select APIs.
  4. On the API Management menu, select Identity Provider. The Identity Provider Overview page is displayed.
  5. Locate the Identity Provider that contains the API key that you need to migrate, then select the pencil (edit) icon.
  6. In the API Clients pane, select the Edit button to display the Identity Provider API key.
  7. Paste the copied Identity Provider API key that you saved in the previous procedure into the key field. Then, click Save.

• NGINX Controller v3 does not support RHEL 7 (11282)

  NGINX Controller v3 does not support Red Hat Enterprise Linux (RHEL) v7. The NGINX Controller installation fails during the Docker version check. Manually updating the Docker version on RHEL 7 does not solve the problem.

Apps and Services

• Certificates imported out of order using drag-and-drop cause an error when creating a cert bundle (10761)

  When certificates are imported using the drag-and-drop feature in the NGINX Controller web interface, the system expects the first certificate that’s added to be the public certificate. Additional certificates that are added are treated as intermediate CA certs.

  If the certificates aren’t added in order, with the public certificate added first, the resulting API request has transposed values for the publicCert and caCert parameters.

  Workaround:

  When adding certificates using drag-and-drop, always drag the public cert file in first, followed by any additional CA certificates. The private key can be dragged in either before or after the certificates. Alternatively, you can choose to copy and paste the PEM text into the corresponding fields.

Platform

• Cannot change SMTP password using the platform/global endpoint (11049)

  Sending the SMTP password in a PATCH request to the platform/global endpoint has no effect.

  Workaround:

  To change the SMTP password, you can use the help.sh script that’s located in /opt/nginx-controller/ on the NGINX Controller host. For instructions, see the documentation that’s installed with NGINX Controller: https://<Controller-FQDN>/docs/platform/using-helper-script/#update-smtp-settings.

Documentation

• Security section appears in the documentation but has no content (11294)

  In NGINX Controller v3.4, a “Security” panel appears in the “Services” section of the onboard documentation. If you click on the Security panel, you will not see any content listed because there is no security documentation for this release. Users should disregard the “Security” panel in the documentation for this release.

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• The NGINX Controller Agent installer script ‘install.sh’ uses HTTP instead of HTTPS to check and install packages (8587)
  • Solution Article: K00958787 | CVE-2020-5867
• The helper.sh script, which is used optionally in NGINX Controller to change settings, uses sensitive items as command-line arguments (7436)
  • Solution Article: K11922628 | CVE-2020-5866
• NGINX Controller uses insecure connection for communications with PostgreSQL database by default (6509)
  • Solution Article: K21009022 | CVE-2020-5865

Additional Fixes

This release includes fixes for the following issues. You can search by the issue ID to locate the details for an issue.

• Creation of eventually consistent objects returns 201 instead of 202 (7052)
• Can’t delete a configured gateway when there is a gateway in an error state (7058)
• Dependencies Missing from Open Source Dependencies Report (7404)
• Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)
• Intermediate certificates not pushed to NGINX Plus instance (7500)
• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)
• Agent warns about failed connection to receiver after update (9152)
• Certificate expiration shows expired warning and expiration date that doesn’t match the final certificate (9167)
• Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)
• Login endpoint has inaccurate session information in the API Reference Guide (9482)
• A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

Apps and Services

• NGINX Controller web interface shows “Error fetching apps” after a restart (10013)

  In some cases, after NGINX Controller restarts, you might see an error in the web interface that says, “Error fetching apps: Network Error.”

  Workaround:

  Log out of the web interface and then log back in.

• Gateways configured with duplicate ingress URIs error out with condition of “config” (10425)

  When upserting a Gateway, if the ingress URI (desiredState.ingress.uris) is a duplicate of another Gateway’s ingress URI, the upsert errors out as expected; however, the condition type (state.conditions.type) that’s reported is simply config, instead of an explanation for the condition that’s occurred.

  "state": {
      "conditions": [
          {
              "type": "config" 
          }
      ],
      "selfConfigState": {
          "configured": 0,
          "configuring": 0,
          "deleting": 0,
          "error": 1,
          "isConfigured": false,
          "isConfiguring": false,
          "isDeleting": false,
          "isError": true,
          "total": 1
      }
  }
  

  Workaround:

  To resolve the error, remove the Gateway with the duplicate ingress URI that’s in the error state. You can do this by issuing a PUT request and applying a new config with a unique ingress URI, or you can issue a DELETE request to remove the Gateway with the duplicate ingress URI.

• Component fails to configure when its Gateway URI collides with another Gateway’s URI (10430)

  In situations where a Component for one Gateway uses the FQDN of a different Gateway on the same host, NGINX Controller doesn’t push the config to NGINX Plus.

  Scenario

  1. Create a Gateway for foo.com with the hostname http://foo.com, and deploy to NGINX Plus ‘Instance 2’.
  2. Create a Component for foo.com with the URI http://foo.com/, and associate the Component with the Gateway foo.com and assign a workload.
  3. Create a Gateway for bar.com with the hostname http://bar.com, and deploy to NGINX Plus ‘Instance 2’.
  4. Create a Component for bar.com with the URI http://foo.com/baz, and associate the Component with the Gateway bar.com and assign a workload.

  In this scenario, the configuration should expose the location for /baz. However, although the Component is reported as configured, nothing happens.

• App Services API Known Issues (10473)

  The App Services API has known issues for certain attributes and objects, in particular:

  • Socket parameters defined in the API should not be used at all. This includes the receivBufferSize and sendBufferSize parameters that are common to Gateway ingress and Component ingress.
  • Session persistence works only for SessionPersistenceCookie. Using SessionPersistenceRoute or SessionPersistenceCookieLearn results in configuration errors.
  • When configuring workload monitoring, and when using the API to check headers in the response, it’s necessary to check for an exact match for the header. Matching with a regular expression or checking only for the existence of a header does not work.
  • A Load Balancing method of HASH won’t work if it contains a variable such as $request_URI for the userKey.

  The existing URI Rewrite capability in the App Services API has the following limitations:

  • The Rewrite directive is not associated with a particular URI or location block in the generated configuration.
  • The break flag is not supported.

Analytics

• SELinux settings can prevent statistics flow from agents (10342)

  Depending on the SELinux settings, an NGINX instance might not have permission to submit statistics to your NGINX Controller.

  If you have a properly configured environment, application, and component for which no metrics are being passed to NGINX Controller after more than 30 seconds of traffic have transpired, try the suggested workaround steps.

  See the NGINX blog post Using NGINX and NGINX Plus with SELinux for additional general guidance.

  Workaround:

  1. On the instance(s) associated with your Gateway, run the following command as root:

     # grep nginx /var/log/audit/audit.log | audit2allow -M nginx`
     

  2. Then, run this command:

     # semodule -i nginx.pp
     

Vulnerability Fixes

Vulnerability issues are disclosed only when a fix is available. For information about a vulnerability fix, including the recommended action, see the linked AskF5 Solution Article for details.

• API allows for unauthenticated account creation (9428)
  • Solution Article: K14631834 | CVE-2020-5863
• NGINX Controller Agent uses insecure connection for communications between NGINX Controller and NGINX Plus instances by default (5899)
  • Solution Article: K27205552 | CVE-2020-5864

Additional Fixes

This release includes fixes for the following issues. You can search by the issue ID to locate the details for an issue.

• Stack trace error occurs when accessing the Login credentials schema in the API reference docs. (6409)
• JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)
• Links to metrics descriptions from user interface point to the metrics reference landing page (7255)
• Usage of API key authentication policies requires the njs module on data-plane instances (7298)
• Omitting URI when defining a component may result in unexpected behavior (7308)
• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

Apps and Services

• Certificate expiration shows expired warning and expiration date that doesn’t match the end-entity certificate (9167)

  A certificate might appear as expired on the certificate detail page, although the Valid To date is in the future. This happens because the summary status and expiration date are calculated based on the entire certificate chain from which the certificate is derived. So, if you have an intermediate CA cert in your chain, the expiration summary status and summary expiration date may reflect the status/date of any intermediate certificate. If you see such a status or expiration date, you should check the status of your entire certificate chain, including that of any intermediate CA.

• A volume of more than 90 pending CUD operations for Gateways and Components could cause NGINX Controller to become unresponsive (9574)

  A high volume of Create, Update, and Delete (CUD) operations – for example, more than 90 – performed on Gateways and Components could create a condition in which NGINX Controller becomes unresponsive.

  NGINX Controller limits queued operations at 100 and then returns an HTTP 429 error: “Too many requests; wait, and try again.” If, subsequently, additional CUD operations are submitted at a rapid rate (more than 1 per second), the queue may be overrun, creating a condition where no further changes can be processed without involvement from NGINX Support.

  Workaround:

  To avoid this situation, limit the amount of API Create, Update, and Delete (CUD) operations sent to the Component and Gateway endpoints to fewer than 100 open actions. The queue should clear 1 action per 7 seconds, on average.

Infrastructure

• Agent warns about failed connection to receiver after update (9152)

  After updating NGINX Controller and the Controller Agent to v3.2.0, the system warns that the Agent “Failed to connect to the receiver.” However, if you look in the agent.log file, the Agent is logging communication with the receiver, as expected.

  This warning occurs when a TLS certificate signed with a custom CA is used for the NGINX Controller. Before v3.2.0, the Controller Agent did not verify server certificates.

  To determine the status of the Agent, perform the following on the Agent host:

  • Check the log file /var/log/nginx-controller/agent.log for errors.

  • Check the status of the Controller Agent service:

    systemctl status controller-agent

    The output should include the status running.

• Agent fails to connect to NGINX Controller if using certs signed with a custom CA (9304)

  If a TLS certificate for NGINX Controller is signed with a custom CA, the Agent fails to verify the cert and returns an error similar to the following:

  2020-03-09 20:38:28,803 [5089] MainThread failed POST "https://<fqdn>:8443/1.4/<uuid>/agent/",
  exception: "HTTPSConnectionPool(host='<fqdn>', port=8443): Max retries exceeded with url: /1.4/<uuid>/agent/
  (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))"
  

  Workaround:

  1. Place the CA on the Agent host.

  2. On the Agent host, edit the /etc/controller-agent/agent.conf file. In the cloud directive section, make the following changes:

     • Add or set the requests_ca_bundle option to point to the CA file.
     • Set verify_ssl_cert to True. (Note: If this setting is set to False, then server certificate validation is disabled, which we don’t recommend.)

     [cloud]
     ...
     verify_ssl_cert = True
     requests_ca_bundle = <path-to-ca-bundle-file>
     

  3. Save the changes to the agent.conf file.

  4. Restart the agent:

     systemctl restart controller-agent

  5. Verify that the Agent can connect to the NGINX Controller host:

     systemctl status controller-agent

     The output should include the status running. You can also check the log file /var/log/nginx-controller/agent.log for errors.

Documentation

• Login endpoint has inaccurate session information in the API Reference Guide (9482)

  The API reference documentation for the Login endpoint describes new session behavior that hasn’t been implemented yet. At this time, the default maximum session lifetime is 8 hours. Once the session lifetime limit is exceeded, you must log in to obtain a new session token.

Installation and Upgrade

• Could not initialize a Kubernetes cluster when installing NGINX Controller on CentOS 7 (5863)

  There is an issue in CentOS 7 that creates a mismatch between the optimum Docker settings and optimum Kubernetes settings. This mismatch prevents Kubernetes from initializing during the installation of NGINX Controller on CentOS 7.

  The NGINX Controller installation fails after Docker and Kubernetes are installed, with the following error: “execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster.”

  Workaround:

  • Solution one:
    • Run sudo yum update. This updates the components to the latest version and patches the issue.
  • Solution two:
    1. Run controller-installer/install.sh and follow the instructions. Allow the installation to fail. This establishes Docker and Kubernetes.
    2. Stop Docker: sudo systemctl stop docker
    3. Edit /etc/docker/daemon.json and make the the following change: native.cgroupdriver=systemd to native.cgroupdriver=cgroupfs
    4. Start Docker: sudo systemctl start docker
    5. Reset the Kubernetes configuration: sudo kubeadm reset -f
    6. Run controller-installer/install.sh and follow the instructions. Rerunning the installation does not update the version of any of the packages on the machine.

Apps and Services

• When committing a gateway with REGEX matchMethod, whose URI includes an explicit port and a path, the error message incorrectly states that the port is invalid (8452)

  In the context of configuring a gateway, only the protocol, domain, and optional port have any meaning. If the URI specified when configuring a REGEX matchMethod gateway includes a path, including a final / after the port, you will receive an error stating: Port value is invalid for hostname .*\\.example\\.com, where the hostname string corresponds to the data you provided for the domain.

  Workaround:

  • Specify your REGEX URI for a gateway as only the protocol, domain, and port with no trailing path.

• REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)

  The logic for interpreting REGEX and REGEX_CASE_SENSITIVE match methods for REGEX-based Component URIs is reversed.

  REGEX-based Component URIs are converted into a case-insensitive location block in nginx.conf when the URI is specified as a relative URI (leads with a path, not a domain) and when the match method is specified as REGEX_CASE_SENSITIVE. Conversely, when the match method is REGEX, the REGEX-based Component URI is interpreted as case sensitive.

  If you need your REGEX-based Component URI to be interpreted as case sensitive, then you can specify the match method as REGEX for now. However, keep in mind that when this logic mismatch is fixed in a later release of NGINX Controller, you’ll need to revert the match method accordingly.

• When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)

  In some cases, when communication between NGINX Controller and the Controller Agent is interrupted, an error similar to the following is returned: “Error: Failed to deploy to instance, Could not find the targeted NGINX running on the system None”. This error neglects to indicate the name of the system or systems for which the communication has failed.

  Workaround:

  • Communication difficulties can be persistent or temporary, depending on the nature of the failure. It is possible to see some more persistent communication failures by looking at the **Infrastructure*- > **Graphics*- pane in NGINX Controller. If the hostname and nginx-plus instance are down, these items will appear with a red background.

  • If all the systems and nginx-plus instances have with a white background, the communication problem could be temporary or intermittent, and won’t necessarily show in the interface.

    Things to check in order are:

    • Ensure the agent host/container is up
    • Ensure the controller-agent process is running
    • Ensure the nginx-plus process is running
    • Verify network communication between NGINX Controller and the Controller Agent on port 443.

Analytics

• Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)

  When upgrading from NGINX Controller 3.0.0 to 3.1.0, the app-centric metrics are not sent from the NGINX instances to NGINX Controller until a new configuration push is performed.

  Workaround:

  • Perform a Put to an object on each NGINX instance to update the config.

Documentation

• API Management module doesn’t need to run on dedicated NGINX Plus instances (8665)

  The API Management documentation states that the API Management module requires a dedicated NGINX Plus instance. This is no longer the case. As of v3.0, the API Management module no longer needs to run on a dedicated instance.

• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

  The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

  The following API actions are considered “experimental” features; use is not supported and functionality may change.

  • Create an Environment
  • Create an App
  • Create or update an App Component with the following properties defined in desiredState:
    • ingress.tcpKeepAlive
    • backend.workloadGroups.uris.failTimeout
    • backend.workloadGroups.uris.resolve
    • backend.workloadGroups.uris.route
    • backend.workloadGroups.uris.service
    • backend.workloadGroups.uris.slowStart
    • backend.workloadGroups.serviceDiscovery
    • backend.workloadGroups.serviceDiscovery.dnsResolver
    • backend.workloadGroups.serviceDiscovery.dnsResolver.uris
    • backend.preserveHostHeader
    • backend.queue
    • backend.httpVersion
    • backend.ntlmAuthentication
    • backend.persistentState
    • programmability.httpHttpsRedirect
    • programmability.uriRedirects
    • programmability.cookieModifications
    • programmability.requestHeaderModifications
    • programmability.responseHeaderModifications
    • security
    • security.waf
    • security.conditionalAuth
    • security.interceptWorkloadErrors
  • Create or update a Gateway with the following properties defined in desiredState:
    • ingress.headers
    • ingress.http2
    • ingress.spdy
    • ingress.proxyProtocol
    • ingress.setFib
    • ingress.fastOpen
    • ingress.backlog
    • ingress.acceptFilter
    • ingress.deferred
    • ingress.isIpv6Only
    • ingress.reusePort
    • ingress.notFoundStatusCode

• Stack trace errors occur in the API reference docs (8481)

  In NGINX Controller, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/.

  For a number of elements, clicking on an object to view the nested content will result in a ReDoc stack trace error.

  Workaround:

  Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.

• Dependencies Missing from Open Source Dependencies Report (7404)

  In NGINX Controller v3.1, the following open source dependencies were omitted from the open source acknowledgements PDF:

  • nats version 2.1.0 (https://github.com/nats-io/nats.c)
  • protobuf version 3.10.1 (https://github.com/protocolbuffers/protobuf)
  • protobuf-c version 1.3.2 (https://github.com/protobuf-c/protobuf-c)

API Management

Analytics

• None

Platform

• Bug Fixes

Load Balancer

• Bug Fixes

API Management

• Preview: Developer Portal:
  
  This release includes a preview of the API Management Module’s Developer Portal feature. Admins can choose to add API documentation – including descriptions, sample requests, and sample responses – for APIs published through Controller. Users can view the API documentation in the preview of the dev portal.

Analytics

• None

Platform

• Bug Fixes

Load Balancer

• Bug fixes

API Management

• JWT Client Groups now available:

  API Management users can choose between using API keys and JWTs for Client Groups. JWT Client Groups allow users to specify a .jwk by providing a URL.

• Client Group API keys can be stored as hashed values:

  API keys can now be stored as SHA-256 hashed values on each API gateway. The JavaScript module njs must be installed on each NGINX Plus instance to enable this feature.

• Updated Default Log Format for API Management:

  The default log format for API Management enables the full range of metrics for graphs, dashboards, and alerts. This resolves a previous issue where publishing an APIM configuration overwrote custom log formats.

Analytics

• This release introduces the new NGINX Controller Analytics module. With an Analytics module license, users can monitor, troubleshoot, and get insights into their NGINX Plus environments. For more information, please reach out to [email protected].

Platform

• Bug Fixes

Load Balancer

• Bug fixes

API Management

• Read-Only Mode for API Management:

  User accounts with the “Read-Only” role are able to view API Management configurations but cannot make any edits.

• Default Proxy Mode:

  HTTP keepalive policies can now be applied to upstream groups. Proxied requests are now made with HTTP/1.1 (previously HTTP/1.0).

• Audit Log:

  The Audit Log lets you view all actions performed by Controller users over a set date range. Audit log results can be sorted and filtered by each column.

• Reusable TLS Policies:

  TLS policies are now managed independently and can be associated with one or more Entry Points. A TLS policy includes the protocols, cipher settings, and session cache settings. The TLS certificate and private key are still Entry Point attributes.

Platform

• Bug fixes

Load Balancer

• Bug fixes

API Management

• Global DNS Resolver: Configure an external resolver that can be optionally enabled within each Upstream Group.

Platform

• New end user license agreement

Load Balancer

• Bug fixes

API Management

• Notifications will appear as you approach and exceed your license limit
• To view license status , click on your API Management license within Settings > Licenses

Platform

• Support contact information is now accessible from User Menu > Support

Load Balancer

• The Instances page is now read-only, displaying the configuration that is currently running on each instance. This will help ensure nobody accidentally overwrites expected policy. To edit instance configurations, you must use Configuration Templates (Load Balancing > Configurations).
  • If an instance is associated to a Load Balancing Configuration Template or an API Management Entry Point, that information is reflected at the bottom of the page.
  • Instances that are not associated with a module can be promoted to Configuration Templates at the bottom of the Instances page for load balancing purposes, or associated with an Entry Point for API Management purposes.
• Configuration Templates > Instances
  • You can now associate instances to a Configuration using tags.
  • Last Modified timestamp now shows the last time the configuration was changed, whether manually or through Controller.

API Management

• Invalid Resource (404) Policy
  • From the Entry Point configuration page, you can specify an alternate HTTP error code, rather than the default 404 response to requests for invalid resources.

Platform

• Configure dashboard graph sets using one or more tags
• Support spaces in upstream group name, route alias, system alias

Load Balancer

• Configuration Templates
  • Ability to preview config file
  • Additional feedback around in-progress config push

API Management

• Ability to edit environment policies
• Usability improvements to API Definitions page
  • Entry point hostname color-coding:
    • Grey - config not pushed to the entry point
    • Green - config pushed, all associated instances are online
    • Yellow - config pushed, one or more instances offline & one or more instances online
    • Red - config pushed, all instances offline
  • Number of unrouted resources is displayed for each environment

Platform

• Ability to configure the ServiceNow alert integration within Controller

Load Balancer

• Configuration Templates - ability to name versions
• Instances are sorted by display name (system alias, or hostname if no alias is set) in the inventory list and dropdown menus
  • Inventory menu in the app header defaults to sorting by display name, with options to change sort parameter

API Management

• New “card” layout for the API Definition list view

Platform

• Ubuntu 18 is qualified for hosting Controller Agents

Load Balancer

• Configuration Templates can now be deleted

Load Balancer

• Configuration Templates
  • Config
  • Associated Instances
  • Version History
  • Version Differencing
  • Arbitrary Reverting
• Multi-push from ‘Load Balancing > Instances’
• Notifications for multi-push status
• ServiceNow alerting integration
• Ability to enable error_log and access_log in virtual server
• Renamed Amplify metrics
• Rename tab from “Configure” to “Load Balancing”
• Ability to tag instances from the ‘Load Balancing > Instances’ tab
• Config-push deferral - ability to change a config and save it without pushing
• Promote session persistence to its own section (Upstream Group config)
• Offline installer / updater
• Various bug-fixes

API Management

• New API Management tab in Controller primary navigation
• Create and manage API Definitions
• Create and manage Upstream Groups
• Create and manage Entry Points
• Manage Clients and Client Groups
• Policies can be applied to various objects in API Management