NGINX Documentation

NGINX Controller Releases

NGINX Controller Version 3.1.0

These release notes provide general information and describe known issues for NGINX Controller version 3.1.0, in the following categories:

For additional release documentation, see the following guides:

Updates

NGINX Controller 3.1 includes the following updates:

  • Bug fixes and improvements.

  • The Components resource adds experimental strategyRef and waf placeholder objects.

    The strategyRef and waf objects for the Components resource are experimental placeholder objects. Including the strategyRef and/or waf blocks in the component specification should not enable additional security-related application traffic services, even if there are no errors. If you see the error “Security module not found on the NGINX instance,” you should remove these blocks.

    Note: The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension.

  • This release adds new app-centric metrics and a new summary page in the UI for viewing these metrics. You can use these app metrics to monitor and evaluate the condition of your apps.

Resolved Issues

This release includes fixes for the following issues. You can search by the issue ID to locate the details for an issue.

  • Cannot add tags to users with the NGINX Controller web interface (7165)
  • Gateway accepts a relative path instead of a URI (7212)
  • A component with no URIs will result in an errored deployment (7311)
  • NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)
  • Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)
  • Referencing a non-existing identity provider for a component does not generate an error (7397)
  • GUI incorrectly shows default error log state as enabled (7447)
  • Component-based access logging format specification is non-functional (7646)
  • Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)
  • Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)
  • API definitions base path is not imported from OAS spec file (7671)
  • The API Reference documentation omits the “Get a Location” endpoint (7747)

Known Issues

Configuration Management

  • When committing a gateway with REGEX matchMethod, whose URI includes an explicit port and a path, the error message incorrectly states that the port is invalid (8452)

    In the context of configuring a gateway, only the protocol, domain, and optional port have any meaning. If the URI specified when configuring a REGEX matchMethod gateway includes a path, including a final / after the port, you will receive an error stating: Port value is invalid for hostname .*\\.example\\.com, where the hostname string corresponds to the data you provided for the domain.

    Workaround:

    • Specify your REGEX URI for a gateway as only the protocol, domain, and port with no trailing path.
  • REGEX and REGEX_CASE_SENSITIVE match method logic is reversed for REGEX-based Component URIs (8455)

    The logic for interpreting REGEX and REGEX_CASE_SENSITIVE match methods for REGEX-based Component URIs is reversed.

    REGEX-based Component URIs are converted into a case-insensitive location block in nginx.conf when the URI is specified as a relative URI (leads with a path, not a domain) and when the match method is specified as REGEX_CASE_SENSITIVE. Conversely, when the match method is REGEX, the REGEX-based Component URI is interpreted as case sensitive.

    If you need your REGEX-based Component URI to be interpreted as case sensitive, then you can specify the match method as REGEX for now. However, keep in mind that when this logic mismatch is fixed in a later release of NGINX Controller, you’ll need to revert the match method accordingly.

  • When there’s a communication error between NGINX Controller and the Controller Agent, the problem system is identified as “None” (8564)

    In some cases, when communication between NGINX Controller and the Controller Agent is interrupted, an error similar to the following is returned: “Error: Failed to deploy to instance, Could not find the targeted NGINX running on the system None”. This error neglects to indicate the name of the system or systems for which the communication has failed.

    Workaround:

    • Communication difficulties can be persistent or temporary, depending on the nature of the failure. It is possible to see some more persistent communication failures by looking at the Infrastructure > Graphics pane in NGINX Controller. If the hostname and nginx-plus instance are down, these items will appear with a red background.

    • If all the systems and nginx-plus instances have with a white background, the communication problem could be temporary or intermittent, and won’t necessarily show in the interface.

      Things to check in order are:

      • Ensure the agent host/container is up
      • Ensure the controller-agent process is running
      • Ensure the nginx-plus process is running
      • Verify network communication between NGINX Controller and the Controller Agent on port 443.

Infrastructure

  • Could not initialize a Kubernetes cluster when installing NGINX Controller on CentOS 7 (5863)

    There is an issue in CentOS 7 that creates a mismatch between the optimum Docker settings and optimum Kubernetes settings. This mismatch prevents Kubernetes from initializing during the installation of NGINX Controller on CentOS 7.

    The NGINX Controller installation fails after Docker and Kubernetes are installed, with the following error: “execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster.”

    Workaround:

    • Solution one:
      • Run sudo yum update. This updates the components to the latest version and patches the issue.
    • Solution two:
      1. Run controller-installer/install.sh and follow the instructions. Allow the installation to fail. This establishes Docker and Kubernetes.
      2. Stop Docker: sudo systemctl stop docker
      3. Edit /etc/docker/daemon.json and make the the following change: native.cgroupdriver=systemd to native.cgroupdriver=cgroupfs
      4. Start Docker: sudo systemctl start docker
      5. Reset the Kubernetes configuration: sudo kubeadm reset -f
      6. Run controller-installer/install.sh and follow the instructions. Rerunning the installation does not update the version of any of the packages on the machine.

Visibility and Analytics

  • Upgrading from Controller 3.0.0 to 3.1.0 breaks application-centric metrics (8669)

    When upgrading from NGINX Controller 3.0.0 to 3.1.0, the app-centric metrics are not sent from the NGINX instances to NGINX Controller until a new configuration push is performed.

    Workaround:

    • Perform a Put to an object on each NGINX instance to update the config.

Documentation

  • API Management module doesn’t need to run on a dedicated NGINX Plus instance (8665)

    The API Management documentation states that the API Management module requires a dedicated NGINX Plus instance. This is no longer the case. As of v3.0, the API Management module no longer needs to run on a dedicated instance.

  • Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

    The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

    The following API actions are considered “experimental” features; use is not supported and functionality may change.

    • Create an Environment
    • Create an App
    • Create or update an App Component with the following properties defined in desiredState:
      • ingress.tcpKeepAlive
      • backend.workloadGroups.uris.failTimeout
      • backend.workloadGroups.uris.resolve
      • backend.workloadGroups.uris.route
      • backend.workloadGroups.uris.service
      • backend.workloadGroups.uris.slowStart
      • backend.workloadGroups.serviceDiscovery
      • backend.workloadGroups.serviceDiscovery.dnsResolver
      • backend.workloadGroups.serviceDiscovery.dnsResolver.uris
      • backend.preserveHostHeader
      • backend.queue
      • backend.httpVersion
      • backend.ntlmAuthentication
      • backend.persistentState
      • programmability.httpHttpsRedirect
      • programmability.uriRedirects
      • programmability.cookieModifications
      • programmability.requestHeaderModifications
      • programmability.responseHeaderModifications
      • security
      • security.waf
      • security.conditionalAuth
      • security.interceptWorkloadErrors
    • Create or update a Gateway with the following properties defined in desiredState:
      • ingress.headers
      • ingress.http2
      • ingress.spdy
      • ingress.proxyProtocol
      • ingress.setFib
      • ingress.fastOpen
      • ingress.backlog
      • ingress.acceptFilter
      • ingress.deferred
      • ingress.isIpv6Only
      • ingress.reusePort
      • ingress.notFoundStatusCode
  • Stack trace errors occur in the API reference docs (8481)

    In NGINX Controller v3.1.0, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/3.1.0.

    For several elements – most notably the login credentials schema – clicking on an object to expand it will result in a ReDoc stack trace error.

    Workaround:

    • Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.
  • Dependencies Missing from Open Source Dependencies Report (7404)

    In NGINX Controller v3.1, the following open source dependencies were omitted from the open source acknowledgements PDF:

    • nats version 2.1.0 (https://github.com/nats-io/nats.c)
    • protobuf version 3.10.1 (https://github.com/protocolbuffers/protobuf)
    • protobuf-c version 1.3.2 (https://github.com/protobuf-c/protobuf-c)

Supported Versions

  • NGINX Plus R19
  • NGINX Plus R20

NGINX Controller Version 3.0.0

These release notes provide general information and describe known issues for NGINX Controller version 3.0.0, in the following categories:

Updates

  • NGINX Controller Upgrade not supported in v3.0.0

    For NGINX Controller v3.0.0, upgrade from previous versions is not supported. Breaking changes were introduced between v2.9 and v3.0.0; as such, the upgrade.sh script is not present in v3.0.0.

  • New look for the web interface and docs

    The NGINX Controller v3.0.0 web interface has a brand-new look that we’re excited to share. Intuitive forms help guide you through your tasks. You’ll notice that features are arranged into four categories to match your workflow:

    • Analytics - View your application health score, alerts, and events
    • Services: Define your apps and their services
    • Infrastructure - Manage your NGINX Plus instances
    • Platform - Manage your NGINX Controller settings and user accounts

    The documentation also has a new look. There’s an API reference guide that documents the REST API endpoints. If you use the web interface, the related documentation is organized by the same categories that you see in the UI so that you can quickly find the topics related to your tasks.

  • Generate Support Package by Using the “helper” Script

    New in NGINX Controller v3.0.0, you can pass the supportpkg argument to the helper.sh script to create a support package. The script gathers NGINX Controller configuration files, logs, and system command output into a tarball. You can use this information to diagnose NGINX Controller issues and/or send the tarball to NGINX Support for troubleshooting assistance. Certs are excluded from the support package.

    Example usage:

    [email protected]:/var/tmp$ /opt/nginx-controller/helper.sh supportpkg
    Gathering System Diagnostics: Please wait ...
    Finished: /var/tmp/supportpkg-20191231T053348MST.tar.gz
    
  • Events API

    This release adds an API for the events that were present in Controller version 2.x.

  • Metrics API

    This release adds an API for the metrics that were present in Controller version 2.x.

Known Issues

API Management

  • Data displayed by the user interface does not match that returned by the REST API for some API Management endpoints (7117)

    When working with API Definitions and Published APIs in the API Management module, the following mismatches between the NGINX Controller user interface and the REST API have been reported. These are dependent on where the user created or modified the affected objects.

    • Identity Provider - API Key: When you create an Identity Provider that uses API Key authentication by using the REST API, the name of the Identity Provider group will not display in the user interface. This behavior resolves if you modify the Identity Provider resource by using the REST API.
    • Identity Provider Client - API Key: When you create an Identity Provider Client that uses API Key authentication by using the user interface, you will receive a “404 - Not Found” error in response to an HTTP GET request for the resource.
    • Published API: When you create a Published API by using the user interface, the Environment, Application, Gateway, Policy, and Routes will be missing from the response to an HTTP GET request for the resource.
    • API Definition: When you create an API Definition by using the REST API, the name of the API Definition is not displayed in the user interface. When you create an API Definition by using the user interface, the URIs will be missing from the response to an HTTP GET request for the resource.
  • Usage of API key authentication policies requires the njs module on data-plane instances (7298)

    In the NGINX Controller documentation for the API Management module, we state that “You must install the njs JavaScript module on each NGINX Plus instance to enable API key hashing” when using API key authentication for an Identity Provider. This information is not accurate.

    You must install the njs module on all NGINX Plus instances if you want to use API key authentication for any element of NGINX Controller (not just for API Management). This enables cryptographically-protected storage of API keys in the NGINX Plus configuration.

    If you do not install the njs module and use API key authentication, whether for API Management or elsewhere, the system may experience errors that are not reported in the user interface. See the NGINX Admin Guide for njs installation instructions.

  • API definitions base path is not imported from OAS spec file (7671)

    After you use the API Management REST API to create an API Definitions from an OpenAPI 3.0 spec file, the base path is empty.

    Workaround:

    • You must manually edit the API Definition and provide the base path for API definitions that are created from imported OAS spec files.
  • JWT and API Key Authentication Policies Cannot Be Combined when Creating an Identity Provider in the user interface (6432)

    Within a single Published API, configuring both an API key authentication policy and a JWT authentication policy results in an error, and you will not be able to publish.

    Workaround:

    • If you’re using the NGINX Controller user interface, only add one type of authentication policy (API Key or JWT) per Published API.
    • If you need to use an API key policy and a JWT policy within a single Published API, you can configure those policies using the external API, on condition that the keys are presented in the header or header/bearer, respectively.

Application Delivery Controller

  • Creation of eventually consistent objects returns 201 instead of 202 (7052)

    Component and gateway objects are considered eventually consistent and will not be immediately configured when created. Currently, these endpoints always return a 201 HTTP code (status created) after a PUT (create) or POST. A 202 HTTP code (status accepted) should be expected instead.

  • User interface cannot be used to correct some error cases from direct API calls (7247)

    A direct API call can result in an errored state that cannot be subsequently corrected via the user interface. For example, when creating a component, if the protocol in the workload URI is missing, the component is created and goes into the isError state. When the user interface is then used to try to add the protocol, the Publish button never becomes green to allow the change to be pushed.

    Workaround:

    • Via the API, use a PUT request to correct the error.
  • A component with no URIs will result in an errored deployment (7311)

    A component must have one or more URIs in its definition. If a component has no URIs, it will result in an error when being deployed, resulting in a failed deployment.

    Workaround:

    • Include one or more URIs in the definition of a component.
  • NGINX Controller generates excessive logs (7131)

  • NGINX Controller generates approximately over 1 million log events (including error, warn, and info) a day.

  • Component-level access log enablement results in component getting stuck in an isConfiguring state (7647)

    When component-level access logging is enabled, either through the web interface or via the public API, publishing the configuration gets stuck in an inConfiguring state. It never advances to a Configured state.

    This condition will block updates to the relevant gateway until the underlying condition with the component is resolved.

    Workaround:

    • Do not enable component-level access logging.

      If you find yourself in this situation, you can perform a PUT using the public API – or republish via the web interface – for the relevant component with component-level access logging disabled. This will result in a configured system and will allow other updates to proceed correctly.

  • Component-based access logging format specification is non-functional (7646)

    When specified via the NGINX Controller public API (it is not available through the web interface), the format string for the component-level access log always errors out with an invalid string format error, regardless of the validity of the string that’s specified.

    Workaround:

    • For components to deploy and successfully be configured, avoid setting the component desiredState.logging.accessLog.format.
  • Gateway ingress URIs must specify either HTTP:// or HTTPS://, otherwise the gateway becomes stuck in Configuring state (7416)

    Gateway ingress URIs must specify either HTTP:// or HTTPS:// as the protocol for the ingress.uri. If the protocol is mistyped in a POST or PUT for a gateway, the gateway can become stuck in a Configuring state that never resolves nor issues an error.

    Workaround:

    • If you mistyped the ingress URI protocol, you can submit another PUT request to change the protocol to HTTP:// or HTTPS:// and the gateway will transition from a Configuring state to a Configured state.
  • Omitting URI when defining a component may result in unexpected behavior (7308)

    When defining components, failure to define the protocol and URI fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

    If you don’t include a URI for one component, and then for a subsequent component you define a URI without designating the protocol, the currentStatus.state.conditions field in the JSON response may not be present for the second component, for which the current state is isError. This makes it challenging to determine the root cause of the issue.

  • REGEX can only be used for path-based URIs and only in components (5747)

    Gateways or components that use match methods of REGEX or REGEX_CASE_SENSITIVE are failing either by erroring out or by getting stuck in a configuring state.

    Ingress URIs using a match method of REGEX or REGEX_CASE_SENSITIVE can be used only in a component (and not in gateways), and will work only when the URIs specify relative paths starting with / or \/.

    For example:

      "uris": {
        "/*.jpg": {
          "matchMethod": "REGEX"
        }
      },
    

    Workaround:

    • If you have a gateway that is unresponsive for more than a few minutes, make sure it doesn’t use REGEX or REGEX_CASE_SENSITIVE as a match method in any of its URIs.
    • If you have a component that is unresponsive, make sure that it complies with the regex guidelines above if REGEX or REGEX_CASE_SENSITIVE is used as a match method.
    • If you need to change the match method to resolve regex issues, you can use the NGINX web interface or send a PUT request using the API to the appropriate gateway or component, and specify a match method of EXACT, PREFIX, or SUFFIX. For a gateway, change your URI string to a string that specifies a full protocol–HTTP or HTTPS–and host. For components, the URI string can specify a relative path; a full protocol and host; or a full protocol and host and path.
  • Web user interface incorrectly shows default error log state as enabled (7447)

    The default setting for the errorLog for a component is Disabled; however, the web interface shows the setting as Enabled when no action has been taken to change it.

    Workaround:

    • Explicitly set the errorLog to the state that is desired, eitherDisabled or Enabled. The web interface will then display an errorLog state that is consistent with what is deployed on the component.
  • Current status does not always give an accurate account of the NGINX Controller configuration (7497)

    Current status does not always reflect the NGINX Controller configuration. This happens, for example, when multiple actions are scripted in a short period (less than 10 seconds) for the same NGINX instance. Examples of inaccurate representations:

    • Components showing that they are in the isConfigured state when the configuration is still being deployed
    • Components don’t show that they are in the isDeleting state while the configuration is being deleted

    Workaround:

    • Avoid scripting multiple actions that run in combination in a short period.
  • Intermediate certificates not pushed to NGINX+ instance (7500)

    When you upload a certificate chain that includes a CA cert, then associate that cert with a gateway or component, only the end-entity certificate (server cert) is pushed to the NGINX+ instance. The intermediate CA certs are not pushed when present. As a result, the chain of trust cannot be established.

    Workaround:

    • Manually deploy certificates and keys to the NGINX+ instance’s local filesystem. For example, copy the key file to NGINX+ at /etc/ssl/mycert.key, and the server cert along with all cacerts in a single file at /etc/ssl/mycert.crt.

      On NGINX Controller, create a cert object of type REMOTE_FILE (use “Add a remote file” in the UI). Use the full path to the files on the NGINX+ instance for privateKey and publicCert.

  • Gateway accepts a relative path instead of a URI (7212)

    When creating or updating a gateway, you must provide a URI as the value for the URI field; URI’s MUST start with a protocol scheme HTTP or HTTPS. If you provide a relative path you will not see any error messages, but NGINX Controller will not write the desired config to the NGINX Plus instance(s).

    Workaround:

    • Do not provide a relative path when defining the URI field for a gateway.

      When defining components, failure to have at least one URI and having URIs with missing protocol fields may result in errors and/or unexpected behaviors. Make sure all components have one or more URIs. Having a component without a URI can have a cascading effect that causes missing conditions fields for other components with errors.

  • Referencing a non-existing identity provider for a component does not generate an error (7397)

    Referencing a non-existing identity provider when defining a component does not generate an error. The status for the component will remain at Configuring.

    Workaround:

    • Remove the reference to the non-existing identity provider.
  • Can’t delete a configured gateway when there is a gateway in an error state (7058)

    When an entity– for example, a gateway, component, or published API–references an instance and is in an error state, requests to DELETE other entities that also reference the instance are accepted but not performed.

    Workaround:

    • Resolve any errors for the entity that references an instance, and then perform the DELETE request on the entity that you want to remove.

General

  • NGINX Controller install.sh shows the incorrect version of Docker that’s installed (7315)

    When installing NGINX Controller, the install.sh script misidentifies the Docker version that’s installed as 19.03.5. NGINX Controller actually installs docker-ce 18.09 and docker-ce-cli 19.03.5. These versions are supported.

    Note: The Docker Command Line Interface (docker-ce-cli) is not an NGINX Controller requirement. Rather, it’s an interface for the administrator to use when working with the Docker Daemon (docker-ce).

  • Cannot delete an instance that doesn’t have NGINX+ installed (7704)

    An instance added to NGINX Controller that doesn’t already have NGINX+ installed cannot be deleted from NGINX Controller.

  • Instance Overview page in the web interface is unresponsive and displays “Loading Instances” (7358)

    The Instance Overview page in the NGINX Controller web interface can become unresponsive if the network interface for any data plane instance is configured only for IPv6. The Instance Overpage page hangs and continually displays the message “Loading Instances.”

  • NGINX Controller returns 429 response to new requests when the queue of unprocessed requests is above threshold (7526)

    NGINX Controller will return a response of 429 “Too many requests” to new PUT, POST, or DELETE requests when its queue depth of unprocessed requests is above a threshold.

    Workaround:

    • Retry the request to which the 429 was received after allowing time for the queue to be processed.
  • Unreachable instance leads to stuck isConfiguring/isDeleting state (7498)

    When a create, update, or delete request is sent to a gateway or component of an instance that cannot be reached, the currentStatus.state.selfConfigState reports true for isConfiguring (for create or update) and true for isDeleting (for delete) indefinitely.

    The state will advance if and only if the instance becomes reachable.

  • Controller Agent stops running on CentOS 6.x (1659)

    When running the Controller Agent on CentOS 6.x the Agent starts and then stops.

    Workaround:

    • Edit the file /etc/controller-agent/agent.conf.
    • Add the lines level = INFO and level = DEBUG to the end of the file. This will keep the Controller Agent stable.
  • Session duration is limited to 8 hours (7239)

    The current session lifetime is limited to a maximum of 8 hours. This limit cannot be configured for longer or shorter durations. Once the session lifetime limit has been exceeded, users must log in again to receive a new session token.

  • Cannot add tags to users with the NGINX Controller web interface (7165)

    In NGINX Controller 3.0.0, you cannot add tags to users when creating or updating users using the web interface, and any tags that are created for a user using the API are not displayed.

  • NGINX Controller registers an existing instance as a new instance when compute VM changes (7844)

    If the compute VM for an NGINX Plus instance changes, NGINX Controller registers the instance as a new instance instead of honoring the existing agent.conf.

Documentation

  • The API Reference documentation omits the “Get a Location” endpoint (7747)

    In the API Reference documentation located at <FQDN/docs/api/ on NGINX Controller, the GET Location endpoint is omitted. The missing information is provided below.

    GET <FQDN>/api/v2/infrastructure/locations/{locationName}
    

    Get a Location:

    • Returns information about a specified Location resource.
    • PATH PARAMETER(S): locationName
      • required
      • string
      • The name of the Location that you want to view.
  • Manage Apps doc incorrectly states that CRUD operation permissions are determined by RBAC (7651)

    In NGINX Controller v3.0.0, the “Manage Apps” document (<FQDN>/docs/services/apps/manage-apps/) contains a line that states “CRUD (Create, Read, Update, Delete) operations on an application are determined by RBAC role authorization.”

    This statement is incorrect. RBAC permissions are not applied at the Apps level in this release. Instead, the ability to perform CRUD operations is granted by the legacy Admin or User Roles.

  • Experimental APIs are not distinguished from supported APIs in the API Reference docs (7734)

    The NGINX Controller REST API may contain API endpoints that are not fully supported. These endpoints are marked with the x-f5-experimental vendor extension. The vendor extensions are not displayed in the on-box API reference documentation.

    Workaround:

    • Download the API spec via the link provided at the top of the API Reference page and view the x-f5-experimental designations in the raw source file.
  • Stack trace errors occur in the API reference docs (6409)

    In NGINX Controller v3.0.0, you can access the API reference documentation in-product via the path http://<controller-FQDN>/docs/api/3.0.0.

    For several elements – most notably the login credentials schema – clicking on an object to expand it will result in a ReDoc stack trace error.

    Workaround:

    • Refresh the page to reload the reference documentation. To view the desired schema, look at the example panel instead of clicking to expand the element.
  • Links to metrics descriptions from user interface point to the metrics reference landing page (7255)

    When you create a new Dashboard using the NGINX Controller user interface, you can click on the “i” (info) symbol to view a description of specific metrics. Clicking the info symbol should bring you to the metrics catalog entry for that metric. In v3.0.0 of NGINX Controller, all of the links point to the metrics reference landing page.

Supported Versions

  • NGINX Plus R19
  • NGINX Plus R20

NGINX Controller Version 2.9.0

These release notes provide general information and describe known issues for NGINX Controller version 2.9.0, in the following categories:

Updates

Analytics

  • None

Platform

  • Bug Fixes

Load Balancer

  • Bug Fixes

API Management

  • Preview: Developer Portal:

    This release includes a preview of the API Management Module’s Developer Portal feature. Admins can choose to add API documentation – including descriptions, sample requests, and sample responses – for APIs published through Controller. Users can view the API documentation in the preview of the dev portal.

Known Issues

  • Kubernetes components that remained configured/running after failed Controller installation prevented reinstall:

    When a Controller installation failed after the kubernetes components were installed, the Kubernetes processes continued to run and configuration was not reset. Subsequent attempts to install the Controller fail because required ports are already in use.

    Workaround:

    1. Check the output of the failed install process and the install log in /var/log/nginx-controller/ to determine why the install process failed. Often, this can occur due to database connectivity or authentication issues.

      You will need to resolve this root cause prior to attempting reinstallation of controller.

    2. When you are ready to reinstall, run the commands shown below and then follow the prompts to remove the Kubernetes configuration.

      sudo kubeadm reset
      mv $HOME/.kube $HOME/.kube-old
      
    3. Proceed with re-installing the Controller.

  • Controller 2.9 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.9 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

  • API Management module allows upload of multiple JWT authentication policies for the same environment:

    When two JWT client group authentication policies are applied to a single environment, publishing to the gateway will fail. Workaround:

    Remove the additional JWT authentication policy, then try to publish the config.

  • Upgrade from 2.6.0 Incorrectly Migrated TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 incorrectly migrated the API Management TLS/SSL Policy resources. Subsequent attempts to edit and publish the APIM config failed.

    Workaround:

    • Create a new TLS/SSL Policy and add it to the affected endpoint.
    • Remove the broken TLS/SSL Policy from the endpoint.
    • Save and publish the APIM config.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file.

    Workaround:

    If you have made manual edits to the agent.conf file, take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps of the New Instance dialog.
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf
    4. Restart the controller-agent process.

Supported Versions

  • NGINX Plus R15
  • NGINX Plus R16
  • NGINX Plus R17
  • NGINX Plus R18
  • NGINX Plus R19

NGINX Controller Version 2.8.1

Resolved Issues

  • Upgrade from Controller v2.7.3 Removed JWT Policy from API Management Config

    The upgrade from Controller v2.7.3 to Controller v2.8.0 removed a JWT policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. This issue is resolved in Controller v2.8.1.

  • Unable to Publish API Management Config with TLS/SSL Policy

    In the Controller-2.8.0 release, attempts to publish an API Management configuration with a TLS/SSL policy failed with an error message indicating that the certificate could not be loaded. This issue is resolved Controller-2.8.1.

Known Issues

  • Upgrade from 2.6.0 Incorrectly Migrated TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 incorrectly migrated the API Management TLS/SSL policy resources. Subsequent attempts to edit and publish the APIM config failed.

    Workaround:

    • Create a new TLS/SSL policy and add it to the affected endpoint.
    • Remove the broken TLS/SSL policy from the endpoint.
    • Save and publish the APIM config.
  • API Management Config Publication Failed After Renaming a JWT Client Group:

    Renaming a client group breaks the link between the client group and its previously published JWT policy. Subsequent attempts to publish the API Management config failed.

    Workaround:

    Do not rename any client groups that are referenced in an active JWT policy.

    If you have already done so, take the following steps:

    1. Delete the policy that references the renamed client group.
    2. Delete the client group.
    3. Create a new client group.
    4. Associate the relevant .jwk file with the new client group.
    5. Configure a JWT policy that references the new client group.
    6. Publish the config.
  • Controller 2.8 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.8 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
    4. Restart the controller-agent process.

NGINX Controller Version 2.8.0

These release notes provide general information and describe known issues for NGINX Controller version 2.8.0, in the following categories:

Updates

  • NGINX Controller runs on Kubernetes:

    This release introduces Kubernetes as an orchestration tool for the NGINX Controller containers.

  • Support for Debian 8 is deprecated in this release.

Analytics

  • None

Platform

  • Bug Fixes

Load Balancer

  • Bug fixes

API Management

  • JWT Client Groups now available:

    API Management users can choose between using API keys and JWTs for Client Groups. JWT Client Groups allow users to specify a .jwk by providing a URL.

  • Client Group API keys can be stored as hashed values:

    API keys can now be stored as SHA-256 hashed values on each API gateway. The JavaScript module njs must be installed on each NGINX Plus instance to enable this feature.

  • Updated Default Log Format for API Management:

    The default log format for API Management enables the full range of metrics for graphs, dashboards, and alerts. This resolves a previous issue where publishing an APIM configuration overwrote custom log formats.

Known Issues

  • NGINX Controller 2.8 requires specific Kubernetes packages to be held at version 1.15.3:

    After installing the Controller, exclude these packages from package upgrades:

    • kubeadm
    • kubectl
    • kubelet
On Centos or Red Hat Enterprise Linux:
  1. Install yum-plugin-versionlock:

    yum install yum-plugin-versionlock

  2. For each package to exclude, run yum versionlock <package name>.

On Debian or Ubuntu:

  1. For each package to exclude, run apt-mark hold <package name>.
  • Swap must be disabled on Controller host prior to install/upgrade:

    Per the Kubernetes requirements, you must disable swap on the host where Kubernetes will run. This must be done prior to installing the Nginx Controller or upgrading from a previous version (2.7.0 or earlier) of the Controller.

    You can find more information about this requirement in the Kubernetes kubeadm documentation.

    Please consult your OS Administrator and/or OS-specific documentation for more information about how to permanently disable swap on your machine.

  • Direct Upgrade from Controller 2.0.0 to Controller 2.8.0 fails:

    If you are using v2.0.0 and follow the upgrade process to install v2.8.0, the process will fail with an error.

    Workaround:

    Upgrade to Controller v2.7.3 first. Then, upgrade to Controller v2.8.0.

  • The API Management module does not support IP addresses when setting the location of a remote .jwk file for JWT Client Groups:

    The API Management module supports the use of JWT for Client Groups beginning in this release. You can either upload a local .jwk file or specify the URL via which a remote file can be accessed. If you provide an IP address when specifying the remote URL, a validation error that states the input is invalid will occur. The remote key file can only be saved if the URL is specified as a fully-qualified domain name (FQDN) .

    Workaround:

    Provide the FQDN and path to the .jwk file instead of using the server IP address.

  • API Management module allows upload of multiple JWT authentication policies for the same environment:

    The API Management module supports the use of JWT for Client Groups beginning in this release. When creating a Client Group, the system will allow you to create more than one JWT authentication policy for a single environment. If two JWT client group authentication policies are applied to a single environment, publishing to the gateway will fail. When you try to publish the config, you will see the following error: “Only one JWT policy can be added to an environment”.

    Workaround:

    Remove the additional JWT authentication policy and then try to publish the config.

  • Upgrade from Controller v2.6.0 Removed TLS/SSL Policy from API Management Config:

    The upgrade from Controller v2.6.0 removed a TLS/SSL policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. Attempts to add a new TLS/SSL policy and then publish the config also failed.

    Workaround:

    Remove, then re-create, the environment that contains the affected policy.

  • Controller 2.8.0 release is incompatible with Controller Agent v2.5.0:

    There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.8.0 to a v2.5.0 Agent fail with a timeout error.

    Workaround:

    Install the Agent by running the commands provided in the “Add new instance” dialog from your agent host.

  • Controller-agent update overwrites agent.conf customizations:

    The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

    1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
    2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
    3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
    4. Restart the controller-agent process.
  • Supported Versions

    • NGINX Plus R15, R16, R17, R18, R19*

    * NGINX Controller 2.8 doesn’t provide GUI support for the status_zone directive in the location block for R19.


    NGINX Controller Version 2.7.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.7.0, in the following categories:

    Updates

    Analytics

    • This release introduces the new NGINX Controller Analytics module. With an Analytics module license, users can monitor, troubleshoot, and get insights into their NGINX Plus environments. For more information, please reach out to [email protected].

    Platform

    • Bug Fixes

    Load Balancer

    • Bug fixes

    API Management

    • Read-Only Mode for API Management:

      User accounts with the “Read-Only” role are able to view API Management configurations but cannot make any edits.

    • Default Proxy Mode:

      HTTP keepalive policies can now be applied to upstream groups. Proxied requests are now made with HTTP/1.1 (previously HTTP/1.0).

    • Audit Log:

      The Audit Log lets you view all actions performed by Controller users over a set date range. Audit log results can be sorted and filtered by each column.

    • Reusable TLS Policies:

      TLS policies are now managed independently and can be associated with one or more Entry Points. A TLS policy includes the protocols, cipher settings, and session cache settings. The TLS certificate and private key are still Entry Point attributes.

    Known Issues

    • Upgrade from Controller v2.6.0 Removed TLS/SSL Policy from API Management Config:

      The upgrade from Controller v2.6.0 removed a TLS/SSL policy from the API Management config. Attempts to publish the APIM config failed due to the missing auth policy. Attempts to add a new TLS/SSL policy and then publish the config also failed.

      Workaround:

      Remove, then re-create, the environment that contains the affected policy.

    • Controller version displayed in UI is v2.7.3:

      For the 2.7.0 release, the Controller UI shows the version number as 2.7.3.

    • Controller 2.7 release is incompatible with Controller Agent v2.5.0:

      There was a non-backwards-compatible change between Agent 2.5.0 and later Agent versions. As a result, API Management configuration publish events from a Controller 2.7 to a v2.5.0 Agent fail with a timeout error.

      Workaround:

      Upgrade the Agent to the latest version by following the instructions provided below to bring the system back into working order.

    • Audit Log’s “Apply” button is disabled when redefining a custom range:

      Steps to reproduce:

      • Create a custom date range on the new Audit Log page and click “Apply”.
      • Make changes to the custom date range.
      • The “Apply” button is not active.

      Workaround:

      • Select and apply a predefined date range, then select a custom range.
    • Upgrade to 2.7.0 doesn’t remove unused Docker container:

      During an update from any earlier version of the controller to v2.7, a previously-used Docker container is left running and emits a warning during the upgrade as follows:

      “Found orphan containers (controller_amplify-extapi_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the –remove-orphans flag to clean it up.”

      In the next release, this container will be cleaned up automatically. To remove it manually in v2.7, you can run the commands shown below:

      sudo docker stop controller_amplify-extapi_1
      sudo docker rm controller_amplify-extapi_1
      
    • Controller-agent update overwrites agent.conf customizations:

      The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

      1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
      2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
      3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
      4. Restart the controller-agent process.

    • Documentation for upgrading the controller agent is out-of-date:

      The instructions provided in the Admin guide and in the in-product documentation contain outdated instructions for upgrading the controller agent.

      To upgrade the agent, you should run the commands provided in the “Add new instance” dialog from your agent host. This will download and run a newer version of the agent.

    • Delay in API initialization post upgrade:

      For a period of about five minutes following updating the controller installation, a generic “Unknown” error banner appears in the UI when accessing the API Management pages. Approximately five minutes following update completion, this self-resolves and the application behaves normally.

      Workaround: You can manually dismiss the “Unknown” error banner.

    • Deletion of API Definition not reflected in GUI until page refresh:

      When you delete an API Definition in the API Module (which is present only if licensed), the deleted definition still appears in the GUI. The definition is, in fact, being deleted correctly. The UI is not updating to reflect this fact.

      Workaround: You can refresh the page, or navigate away and then return, to view the correct state of the application.

    Supported Versions

    • NGINX Plus R15, R16, R17, R18

    NGINX Controller Version 2.6.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.6.0, in the following categories:

    Updates

    Platform

    • Bug fixes

    Load Balancer

    • Bug fixes

    API Management

    • Global DNS Resolver: Configure an external resolver that can be optionally enabled within each Upstream Group.

    Known Issues

    • API Management policies applied may inherit a broader scope than intended.

      To resolve this issue, upgrade to Controller version 2.7.

    • Controller-agent update overwrites agent.conf customizations:

      The controller-agent update process does not respect any manual edits you might have made to the agent.conf file. If you have made manual edits to the agent.conf file, please take the following steps when you update the agent:

      1. Backup your current agent.conf file prior to executing the agent upgrade procedure.
      2. Perform the upgrade by reapplying the steps in the “Add new instance” dialog (as noted below).
      3. Restore your agent.conf file over the file in /etc/controller-agent/agent.conf.
      4. Restart the controller-agent process.
    • Documentation for upgrading the controller agent is out-of-date:

      The instructions provided in the Admin guide and in the in-product documentation contain outdated instructions for upgrading the controller agent.

      To upgrade the agent, you should run the commands provided in the “Add new instance” dialog from your agent host. This will download and run a newer version of the agent.

    • API Management returns 504 Gateway Timeout for a period of minutes following upgrade from 2.5.0 to 2.6.0:

      For a period of about five minutes following updating the controller installation, a generic “Unknown” error appear in the GUI when accessing the API Management pages. Approximately five minutes following update completion, this self-resolves and the application behaves normally.

      Note that the “Unknown” error, which presents as a banner in the UI, must be manually dismissed.

    • Deleted API definition remains on the page:

      When you delete an API Definition in the API Module (present only if licensed), the deleted definition still appears in the GUI. The definition is, in fact, being deleted correctly. The UI is not updating to reflect this fact. You can refresh the page or navigate away and then return to show the correct state of the application.

    Supported Versions

    • NGINX Plus R15, R16, R17, R18

    NGINX Controller Version 2.5.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.5.0, in the following categories:

    Updates

    Platform

    • New end user license agreement

    Load Balancer

    • Bug fixes

    API Management

    • Notifications will appear as you approach and exceed your license limit
    • To view license status , click on your API Management license within Settings > Licenses

    Known Issues

    • Clients deploying controller-agent on CentOS 6 need to manually modify the logging level of controller-agent to ERROR level from INFO and restart controller-agent.
    • Administrators who delete a JWT policy will still see that policy in the environment select list. They should not select/configure a deleted policy.
    • NGINX Plus instances managed by the API Management module must be associated with a single Entry Point
    • Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.

    Supported Versions

    • NGINX Plus R15, R16, R17, R18

    NGINX Controller Version 2.4.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.4.0, in the following categories:

    Updates

    Platform

    • Support contact information is now accessible from User Menu > Support

    Load Balancer

    • The Instances page is now read-only, displaying the configuration that is currently running on each instance. This will help ensure nobody accidentally overwrites expected policy. To edit instance configurations, you must use Configuration Templates (Load Balancing > Configurations).
      • If an instance is associated to a Load Balancing Configuration Template or an API Management Entry Point, that information is reflected at the bottom of the page.
      • Instances that are not associated with a module can be promoted to Configuration Templates at the bottom of the Instances page for load balancing purposes, or associated with an Entry Point for API Management purposes.
    • Configuration Templates > Instances
      • You can now associate instances to a Configuration using tags.
      • Last Modified timestamp now shows the last time the configuration was changed, whether manually or through Controller.

    API Management

    • Invalid Resource (404) Policy
      • From the Entry Point configuration page, you can specify an alternate HTTP error code, rather than the default 404 response to requests for invalid resources.

    Known Issues

    • Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.
    • NGINX Plus instances managed by the API Management module must be associated with a single Entry Point.

    Supported Versions

    • NGINX Plus R15, R16, R17, R18

    NGINX Controller Version 2.3.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.3.0, in the following categories:

    Updates

    Platform

    • Configure dashboard graph sets using one or more tags
    • Support spaces in upstream group name, route alias, system alias

    Load Balancer

    • Configuration Templates
      • Ability to preview config file
      • Additional feedback around in-progress config push

    API Management

    • Ability to edit environment policies
    • Usability improvements to API Definitions page
      • Entry point hostname color-coding:
        • Grey - config not pushed to the entry point
        • Green - config pushed, all associated instances are online
        • Yellow - config pushed, one or more instances offline & one or more instances online
        • Red - config pushed, all instances offline
      • Number of unrouted resources is displayed for each environment

    Known Issues

    • Amazon Linux 2. Docker installation issues: Controller install script cannot handle the setup of docker. Please install docker before running the controller install script.

    Supported Versions

    • NGINX Plus R15, R16, R17, R18

    NGINX Controller Version 2.2.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.2.0, in the following categories:

    Updates

    Platform

    • Ability to configure the ServiceNow alert integration within Controller

    Load Balancer

    • Configuration Templates - ability to name versions
    • Instances are sorted by display name (system alias, or hostname if no alias is set) in the inventory list and dropdown menus
      • Inventory menu in the app header defaults to sorting by display name, with options to change sort parameter

    API Management

    • New “card” layout for the API Definition list view

    Known Issues

    No known major issues in this release

    Supported Versions

    • NGINX Plus R15, R16, R17

    NGINX Controller Version 2.1.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.1.0, in the following categories:

    Updates

    Platform

    • Ubuntu 18 is qualified for hosting Controller Agents

    Load Balancer

    • Configuration Templates can now be deleted

    Known Issues

    No known major issues in this release


    NGINX Controller Version 2.0.0

    These release notes provide general information and describe known issues for NGINX Controller version 2.0.0, in the following categories:

    Updates

    Load Balancer

    • Configuration Templates
      • Config
      • Associated Instances
      • Version History
      • Version Differencing
      • Arbitrary Reverting
    • Multi-push from ‘Load Balancing > Instances’
    • Notifications for multi-push status
    • ServiceNow alerting integration
    • Ability to enable error_log and access_log in virtual server
    • Renamed Amplify metrics
    • Rename tab from “Configure” to “Load Balancing”
    • Ability to tag instances from the ‘Load Balancing > Instances’ tab
    • Config-push deferral - ability to change a config and save it without pushing
    • Promote session persistence to its own section (Upstream Group config)
    • Offline installer / updater
    • Various bug-fixes

    API Management

    • New API Management tab in Controller primary navigation
    • Create and manage API Definitions
    • Create and manage Upstream Groups
    • Create and manage Entry Points
    • Manage Clients and Client Groups
    • Policies can be applied to various objects in API Management

    Known Issues

    • If upgrading from a previous version of Controller, password needs to be updated/reset.
    • Load Balancing - Instances page hangs when there are no hosts registered in Controller.
    • Load Balancing - Orange indicators will appear when you have unsaved changes, and may take several minutes to accurately reflect the correct status.
    • API Management - When configuring multiple rate limit policies for an Environment, each must use the same status code.

    NGINX Controller Version 1.0.1

    These release notes provide general information and describe known issues for NGINX Controller version 1.0.1, in the following categories:

    Updates

    • Alerts - fix issue where editing an alert with threshold could result in an error
    • Analyzer - fix issue where old SSL cert could still be evident in the report alongside the new cert that replaced it
    • Analyzer - show appropriate message when port is missing for the listen directive
    • Configure - provide better error descriptions in the UI
    • Configure - fix issue where error notifications disappeared too quickly before they could be fully read
    • Configure - fix issue where user would see “undefined” error when loading a config that included a stream block
    • Configure - fix issue where virtual server SSL block is inactive after a page reload
    • Configure - fix issue where empty directives were not copied to the generated config file
    • Configure - fix issue where proxy is not allowed in regex location
    • Configure - fix issue where the validation for healthcheck URI does not allow for query params
    • Configure - fix issue of no validation for duplicate upstreams
    • Configure - fix the wrong visualisation of healthcheck config after a push config action fails
    • Configure - fix error about server name being too long in the generated config

    Known Issues

    No known major issues in this release