NGINX Controller Releases

Updates

NGINX Controller 3.1 includes the following updates:

• Bug fixes and improvements.

• The Components resource adds experimental strategyRef and waf placeholder objects.

  The strategyRef and waf objects for the Components resource are experimental placeholder objects. Including the strategyRef and/or waf blocks in the component specification should not enable additional security-related application traffic services, even if there are no errors. If you see the error “Security module not found on the NGINX instance,” you should remove these blocks.

  Note: The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension.

• This release adds new app-centric metrics and a new summary page in the UI for viewing these metrics. You can use these app metrics to monitor and evaluate the condition of your apps.

Resolved Issues

This release includes fixes for the following issues. You can search by the issue ID to locate the details for an issue.

• Cannot add tags to users with the NGINX Controller web interface (7165)
• Gateway accepts a relative path instead of a URI (7212)
• A component with no URIs will result in an errored deployment (7311)
• NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)
• Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)
• Referencing a non-existing identity provider for a component does not generate an error (7397)
• GUI incorrectly shows default error log state as enabled (7447)
• Component-based access logging format specification is non-functional (7646)
• Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)
• Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)
• API definitions base path is not imported from OAS spec file (7671)
• The API Reference documentation omits the “Get a Location” endpoint (7747)

Known Issues

Supported Versions

• NGINX Plus R19
• NGINX Plus R20

Updates

• NGINX Controller Upgrade not supported in v3.0.0

  For NGINX Controller v3.0.0, upgrade from previous versions is not supported. Breaking changes were introduced between v2.9 and v3.0.0; as such, the upgrade.sh script is not present in v3.0.0.

• New look for the web interface and docs

  The NGINX Controller v3.0.0 web interface has a brand-new look that we’re excited to share. Intuitive forms help guide you through your tasks. You’ll notice that features are arranged into four categories to match your workflow:

  • Analytics - View your application health score, alerts, and events
  • Services: Define your apps and their services
  • Infrastructure - Manage your NGINX Plus instances
  • Platform - Manage your NGINX Controller settings and user accounts

  The documentation also has a new look. There’s an API reference guide that documents the REST API endpoints. If you use the web interface, the related documentation is organized by the same categories that you see in the UI so that you can quickly find the topics related to your tasks.

• Generate Support Package by Using the “helper” Script

  New in NGINX Controller v3.0.0, you can pass the supportpkg argument to the helper.sh script to create a support package. The script gathers NGINX Controller configuration files, logs, and system command output into a tarball. You can use this information to diagnose NGINX Controller issues and/or send the tarball to NGINX Support for troubleshooting assistance. Certs are excluded from the support package.

  Example usage:

  [email protected]:/var/tmp$ /opt/nginx-controller/helper.sh supportpkg
  Gathering System Diagnostics: Please wait ...
  Finished: /var/tmp/supportpkg-20191231T053348MST.tar.gz
  

• Events API

  This release adds an API for the events that were present in Controller version 2.x.

• Metrics API

  This release adds an API for the metrics that were present in Controller version 2.x.

Known Issues

Application Delivery Controller

• Creation of eventually consistent objects returns 201 instead of 202 (7052)

  Component and gateway objects are considered eventually consistent and will not be immediately configured when created. Currently, these endpoints always return a 201 HTTP code (status created) after a PUT (create) or POST. A 202 HTTP code (status accepted) should be expected instead.

• User interface cannot be used to correct some error cases from direct API calls (7247)

  A direct API call can result in an errored state that cannot be subsequently corrected via the user interface. For example, when creating a component, if the protocol in the workload URI is missing, the component is created and goes into the isError state. When the user interface is then used to try to add the protocol, the Publish button never becomes green to allow the change to be pushed.

  Workaround:

  • Via the API, use a PUT request to correct the error.

• A component with no URIs will result in an errored deployment (7311)

  A component must have one or more URIs in its definition. If a component has no URIs, it will result in an error when being deployed, resulting in a failed deployment.

  Workaround:

  • Include one or more URIs in the definition of a component.

• NGINX Controller generates excessive logs (7131)

• NGINX Controller generates approximately over 1 million log events (including error, warn, and info) a day.

• Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)

  When component-level access logging is enabled, either through the web interface or via the public API, publishing the configuration gets stuck in an inConfiguring state. It never advances to a Configured state.

  This condition will block updates to the relevant gateway until the underlying condition with the component is resolved.

  Workaround:

  • Do not enable component-level access logging.

    If you find yourself in this situation, you can perform a PUT using the public API – or republish via the web interface – for the relevant component with component-level access logging disabled. This will result in a configured system and will allow other updates to proceed correctly.

• Component-based access logging format specification is non-functional (7646)

  When specified via the NGINX Controller public API (it is not available through the web interface), the format string for the component-level access log always errors out with an invalid string format error, regardless of the validity of the string that’s specified.

  Workaround:

  • For components to deploy and successfully be configured, avoid setting the component desiredState.logging.accessLog.format.

• Gateway ingress URIs must specify either HTTP:// or HTTPS://, otherwise the gateway becomes stuck in Configuring state (7416)

  Gateway ingress URIs must specify either HTTP:// or HTTPS:// as the protocol for the ingress.uri. If the protocol is mistyped in a POST or PUT for a gateway, the gateway can become stuck in a Configuring state that never resolves nor issues an error.

  Workaround:

  • If you mistyped the ingress URI protocol, you can submit another PUT request to change the protocol to HTTP:// or HTTPS:// and the gateway will transition from a Configuring state to a Configured state.

• Omitting URI when defining a component may result in unexpected behavior (7308)

  When defining components, failure to define the protocol and URI fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

  If you don’t include a URI for one component, and then for a subsequent component you define a URI without designating the protocol, the currentStatus.state.conditions field in the JSON response may not be present for the second component, for which the current state is isError. This makes it challenging to determine the root cause of the issue.

• REGEX can only be used for path-based URIs and only in components (5747)

  Gateways or components that use match methods of REGEX or REGEX_CASE_SENSITIVE are failing either by erroring out or by getting stuck in a configuring state.

  Ingress URIs using a match method of REGEX or REGEX_CASE_SENSITIVE can be used only in a component (and not in gateways), and will work only when the URIs specify relative paths starting with / or \/.

  For example:

    "uris": {
      "/*.jpg": {
        "matchMethod": "REGEX"
      }
    },
  

  Workaround:

  • If you have a gateway that is unresponsive for more than a few minutes, make sure it doesn’t use REGEX or REGEX_CASE_SENSITIVE as a match method in any of its URIs.
  • If you have a component that is unresponsive, make sure that it complies with the regex guidelines above if REGEX or REGEX_CASE_SENSITIVE is used as a match method.
  • If you need to change the match method to resolve regex issues, you can use the NGINX web interface or send a PUT request using the API to the appropriate gateway or component, and specify a match method of EXACT, PREFIX, or SUFFIX. For a gateway, change your URI string to a string that specifies a full protocol–HTTP or HTTPS–and host. For components, the URI string can specify a relative path; a full protocol and host; or a full protocol and host and path.

• Web user interface incorrectly shows default error log state as enabled (7447)

  The default setting for the errorLog for a component is Disabled; however, the web interface shows the setting as Enabled when no action has been taken to change it.

  Workaround:

  • Explicitly set the errorLog to the state that is desired, eitherDisabled or Enabled. The web interface will then display an errorLog state that is consistent with what is deployed on the component.

• Current status does not always give an accurate account of the NGINX Controller configuration (7497)

  Current status does not always reflect the NGINX Controller configuration. This happens, for example, when multiple actions are scripted in a short period (less than 10 seconds) for the same NGINX instance. Examples of inaccurate representations:

  • Components showing that they are in the isConfigured state when the configuration is still being deployed
  • Components don’t show that they are in the isDeleting state while the configuration is being deleted

  Workaround:

  • Avoid scripting multiple actions that run in combination in a short period.

• Intermediate certificates not pushed to NGINX+ instance (7500)

  When you upload a certificate chain that includes a CA cert, then associate that cert with a gateway or component, only the end-entity certificate (server cert) is pushed to the NGINX+ instance. The intermediate CA certs are not pushed when present. As a result, the chain of trust cannot be established.

  Workaround:

  • Manually deploy certificates and keys to the NGINX+ instance’s local filesystem. For example, copy the key file to NGINX+ at /etc/ssl/mycert.key, and the server cert along with all cacerts in a single file at /etc/ssl/mycert.crt.

    On NGINX Controller, create a cert object of type REMOTE_FILE (use “Add a remote file” in the UI). Use the full path to the files on the NGINX+ instance for privateKey and publicCert.

• Gateway accepts a relative path instead of a URI (7212)

  When creating or updating a gateway, you must provide a URI as the value for the URI field; URI’s MUST start with a protocol scheme HTTP or HTTPS. If you provide a relative path you will not see any error messages, but NGINX Controller will not write the desired config to the NGINX Plus instance(s).

  Workaround:

  • Do not provide a relative path when defining the URI field for a gateway.

    When defining components, failure to have at least one URI and having URIs with missing protocol fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

• Referencing a non-existing identity provider for a component does not generate an error (7397)

  Referencing a non-existing identity provider when defining a component does not generate an error. The status for the component will remain at Configuring.

  Workaround:

  • Remove the reference to the non-existing identity provider.

• Can’t delete a configured gateway when there is a gateway in an error state (7058)

  When an entity– for example, a gateway, component, or published API–references an instance and is in an error state, requests to DELETE other entities that also reference the instance are accepted but not performed.

  Workaround:

  • Resolve any errors for the entity that references an instance, and then perform the DELETE request on the entity that you want to remove.

General

• NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)

  When installing NGINX Controller, the install.sh script misidentifies the Docker version that’s installed as 19.03.5. NGINX Controller actually installs docker-ce 18.09 and docker-ce-cli 19.03.5. These versions are supported.

  Note: The Docker Command Line Interface (docker-ce-cli) is not an NGINX Controller requirement. Rather, it’s an interface for the administrator to use when working with the Docker Daemon (docker-ce).

• Cannot delete an instance that doesn’t have NGINX+ installed (7704)

  An instance added to NGINX Controller that doesn’t already have NGINX+ installed cannot be deleted from NGINX Controller.

• Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)

  The Instance Overview page in the NGINX Controller web interface can become unresponsive if the network interface for any data plane instance is configured only for IPv6. The Instance Overpage page hangs and continually displays the message “Loading Instances.”

• NGINX Controller returns 429 response to new requests when the queue of unprocessed requests is above threshold (7526)

  NGINX Controller will return a response of 429 “Too many requests” to new PUT, POST, or DELETE requests when its queue depth of unprocessed requests is above a threshold.

  Workaround:

  • Retry the request to which the 429 was received after allowing time for the queue to be processed.

• Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)

  When a create, update, or delete request is sent to a gateway or component of an instance that cannot be reached, the currentStatus.state.selfConfigState reports true for isConfiguring (for create or update) and true for isDeleting (for delete) indefinitely.

  The state will advance if and only if the instance becomes reachable.

• Controller Agent stops running on CentOS 6.x (1659)

  When running the Controller Agent on CentOS 6.x the Agent starts and then stops.

  Workaround:

  • Edit the file /etc/controller-agent/agent.conf.
  • Add the lines level = INFO and level = DEBUG to the end of the file. This will keep the Controller Agent stable.

• Session duration is limited to 8 hours (7239)

  The current session lifetime is limited to a maximum of 8 hours. This limit cannot be configured for longer or shorter durations. Once the session lifetime limit has been exceeded, users must log in again to receive a new session token.

• Cannot add tags to users with the NGINX Controller web interface (7165)

  In NGINX Controller 3.0.0, you cannot add tags to users when creating or updating users using the web interface, and any tags that are created for a user using the API are not displayed.

• NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)

  If the compute VM for an NGINX Plus instance changes, NGINX Controller registers the instance as a new instance instead of honoring the existing agent.conf.

Documentation

• The API Reference documentation omits the “Get a Location” endpoint (7747)

  In the API Reference documentation located at <FQDN/docs/api/ on NGINX Controller, the GET Location endpoint is omitted. The missing information is provided below.

  GET <FQDN>/api/v2/infrastructure/locations/{locationName}
  

  Get a Location:

  • Returns information about a specified Location resource.
  • PATH PARAMETER(S): locationName
    • required
    • string
    • The name of the Location that you want to view.

• Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)

  In NGINX Controller v3.0.0, the “Manage Apps” document (<FQDN>/docs/services/apps/manage-apps/) contains a line that states “CRUD (Create, Read, Update, Delete) operations on an application are determined by RBAC role authorization.”

  This statement is incorrect. RBAC permissions are not applied at the Apps level in this release. Instead, the ability to perform CRUD operations is granted by the legacy Admin or User Roles.

• Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

  The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

  Workaround:

  • Download the API spec via the link provided at the top of the API Reference page and view the x-f5-experimental designations in the raw source file.

• Stack trace errors occur in the API reference docs (6409)

  In NGINX Controller v3.0.0, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/3.0.0.

  For several elements – most notably the login credentials schema – clicking on an object to expand it will result in a ReDoc stack trace error.

  Workaround:

  • Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.

• Links to metrics descriptions from user interface point to the metrics reference landing page (7255)

  When you create a new Dashboard using the NGINX Controller user interface, you can click on the “i” (info) symbol to view a description of specific metrics. Clicking the info symbol should bring you to the metrics catalog entry for that metric. In v3.0.0 of NGINX Controller, all of the links point to the metrics reference landing page.

Supported Versions

• NGINX Plus R19
• NGINX Plus R20

Updates

Known Issues

• Kubernetes components that remained configured/running after failed Controller installation prevented reinstall:
  
  When a Controller installation failed after the kubernetes components were installed, the Kubernetes processes continued to run and configuration was not reset. Subsequent attempts to install the Controller fail because required ports are already in use.
  
  Workaround:
  
  

  1. Check the output of the failed install process and the install log in /var/log/nginx-controller/ to determine why the install process failed. Often, this can occur due to database connectivity or authentication issues.

     You will need to resolve this root cause prior to attempting reinstallation of controller.

  2. When you are ready to reinstall, run the commands shown below and then follow the prompts to remove the Kubernetes configuration.

     sudo kubeadm reset
     mv $HOME/.kube $HOME/.kube-old
     

  3. Proceed with re-installing the Controller.
     
     

• Controller 2.9 release is incompatible with Controller Agent v2.5.0:
  
  There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.9 to a v2.5.0 Agent fail with a timeout error.
  
  Workaround:
  
  Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

• API Management module allows upload of multiple JWT authentication policies for the same environment:
  
  When two JWT client group authentication policies are applied to a single environment, publishing to the gateway will fail. Workaround:
  
  Remove the additional JWT authentication policy, then try to publish the config.

• Upgrade from 2.6.0 Incorrectly Migrated TLS/SSL Policy from API Management Config:
  
  The upgrade from Controller v2.6.0 incorrectly migrated the API Management TLS/SSL Policy resources. Subsequent attempts to edit and publish the APIM config failed.

  Workaround:
  
  

  • Create a new TLS/SSL Policy and add it to the affected endpoint.
  • Remove the broken TLS/SSL Policy from the endpoint.
  • Save and publish the APIM config.
    
    

• Controller-agent update overwrites agent.conf customizations:
  
  The controller-agent update process does not respect any manual edits you might have made to the agent.conf file.

  Workaround:
  
  If you have made manual edits to the agent.conf file, take the following steps when you update the agent:

  1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
  2. Perform the upgrade by reapplying the steps of the New Instance dialog.
  3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf
  4. Restart the controller-agent process.

Supported Versions

• NGINX Plus R15
• NGINX Plus R16
• NGINX Plus R17
• NGINX Plus R18
• NGINX Plus R19

Updates

• NGINX Controller runs on Kubernetes:

  This release introduces Kubernetes as an orchestration tool for the NGINX Controller containers.

• Support for Debian 8 is deprecated in this release.

Known Issues

• NGINX Controller 2.8 requires specific Kubernetes packages to be held at version 1.15.3:

  After installing the Controller, exclude these packages from package upgrades:

  • kubeadm
  • kubectl
  • kubelet

Supported Versions

• NGINX Plus R15, R16, R17, R18, R19*

* NGINX Controller 2.8 doesn’t provide GUI support for the status_zone directive in the location block for R19.

Updates

Known Issues

• Upgrade from Controller v2.6.0 Removed TLS/SSL Policy from API Management Config:

  The upgrade from Controller v2.6.0 removed a TLS/SSL policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. Attempts to add a new TLS/SSL policy and then publish the config also failed.

  Workaround:

  Remove, then re-create, the environment that contains the affected policy.

• Controller version displayed in UI is v2.7.3:

  For the 2.7.0 release, the Controller UI shows the version number as 2.7.3.

• Controller 2.7 release is incompatible with Controller Agent v2.5.0:

  There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.7 to a v2.5.0 Agent fail with a timeout error.

  Workaround:

  Upgrade the Agent to the latest version by following the instructions provided below to bring the system back into working order.

• Audit Log’s “Apply” button is disabled when redefining a custom range:

  Steps to reproduce:

  • Create a custom date range on the new Audit Log page and click “Apply”.
  • Make changes to the custom date range.
  • The “Apply” button is not active.

  Workaround:

  • Select and apply a predefined date range, then select a custom range.

• Upgrade to 2.7.0 doesn’t remove unused Docker container:

  During an update from any earlier version of the controller to v2.7, a previously-used Docker container is left running and emits a warning during the upgrade as follows:

  “Found orphan containers (controller_amplify-extapi_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the –remove-orphans flag to clean it up.”

  In the next release, this container will be cleaned up automatically. To remove it manually in v2.7, you can run the commands shown below:

  sudo docker stop controller_amplify-extapi_1
  sudo docker rm controller_amplify-extapi_1
  

• Controller-agent update overwrites agent.conf customizations:

  The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

  1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
  2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
  3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
  4. Restart the controller-agent process.

Supported Versions

• NGINX Plus R15, R16, R17, R18

Updates

Known Issues

• API Management policies applied may inherit a broader scope than intended.

  To resolve this issue, upgrade to Controller version 2.7.

• Controller-agent update overwrites agent.conf customizations:

  The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

  1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
  2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
  3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
  4. Restart the controller-agent process.

• Documentation for upgrading the controller agent is out-of-date:

  The instructions provided in the Admin guide and in the in-product documentation contain outdated instructions for upgrading the controller agent.

  To upgrade the agent, you should run the commands provided in the “Add new instance” dialog from your agent host. This will download and run a newer version of the agent.

• API Management returns 504 Gateway Timeout for a period of minutes following upgrade from 2.5.0 to 2.6.0:

  For a period of about five minutes following updating the controller installation, a generic “Unknown” error appear in the GUI when accessing the API Management pages. Approximately five minutes following update completion, this self-resolves and the application behaves normally.

  Note that the “Unknown” error, which presents as a banner in the UI, must be manually dismissed.

• Deleted API definition remains on the page:

  When you delete an API Definition in the API Module (present only if licensed), the deleted definition still appears in the GUI. The definition is, in fact, being deleted correctly. The UI is not updating to reflect this fact. You can refresh the page or navigate away and then return to show the correct state of the application.

Supported Versions

• NGINX Plus R15, R16, R17, R18

Updates

Known Issues

• Clients deploying controller-agent on CentOS 6 need to manually modify the logging level of controller-agent to ERROR level from INFO and restart controller-agent.
• Administrators who delete a JWT policy will still see that policy in the environment select list. They should not select/configure a deleted policy.
• NGINX Plus instances managed by the API Management module must be associated with a single Entry Point
• Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.

Supported Versions

• NGINX Plus R15, R16, R17, R18

Updates

Known Issues

• Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.
• NGINX Plus instances managed by the API Management module must be associated with a single Entry Point.

Supported Versions

• NGINX Plus R15, R16, R17, R18

Updates

Known Issues

• Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.

Supported Versions

• NGINX Plus R15, R16, R17, R18

Updates

Known Issues

No known major issues in this release

Supported Versions

• NGINX Plus R15, R16, R17

Updates

Known Issues

No known major issues in this release

Updates

Known Issues

• If upgrading from a previous version of Controller, password needs to be updated/reset.
• Load Balancing - Instances page hangs when there are no hosts registered in Controller.
• Load Balancing - Orange indicators will appear when you have unsaved changes, and may take several minutes to accurately reflect the correct status.
• API Management - When configuring multiple rate limit policies for an Environment, each must use the same status code.

Updates

• Alerts - fix issue where editing an alert with threshold could result in an error
• Analyzer - fix issue where old SSL cert could still be evident in the report alongside the new cert that replaced it
• Analyzer - show appropriate message when port is missing for the listen directive
• Configure - provide better error descriptions in the UI
• Configure - fix issue where error notifications disappeared too quickly before they could be fully read
• Configure - fix issue where user would see “undefined” error when loading a config that included a stream block
• Configure - fix issue where virtual server SSL block is inactive after a page reload
• Configure - fix issue where empty directives were not copied to the generated config file
• Configure - fix issue where proxy is not allowed in regex location
• Configure - fix issue where the validation for healthcheck URI does not allow for query params
• Configure - fix issue of no validation for duplicate upstreams
• Configure - fix the wrong visualisation of healthcheck config after a push config action fails
• Configure - fix error about server name being too long in the generated config

Known Issues

No known major issues in this release