Known Issues

This document lists and describes the known issues and possible workarounds in NGINX Management Suite Instance Manager. Fixed issues are removed after 45 days.

Tip:
We recommend you upgrade to the latest version of Instance Manager to take advantage of new features, improvements, and bug fixes.

2.6.0

Large payloads can result in disk I/O error for database operations

Issue ID Status
38827 Open

Description

When NGINX Management Suite is storing NGINX config with references to large payloads, a “disk I/O error” may occur. This affects users with config payload sizes larger than 5M, such as the NGINX App Protect policy bundle.


Policy API endpoint only allows NGINX App Protect policy Upsert with content length upto 3.14MB

Issue ID Status
38839 Open

Description

The Policy API endpoint prevents App Protect creation or updates with content-length greater than 3.14MB.


Publishing to an Instance/instance-group might fail when the configuration references a JSON policy or a JSON log profile

Issue ID Status
38357 Open

Description

Publishing to an Instance/instance-group will fail if nginx.conf references a JSON policy or a JSON log profile. NGINX App Protect directives that reference a Security Policy or a Log Profile expect a bundle (.tgz), not a JSON file.

For example, a configuration similar to the following will fail:

location / {    
    app_protect_enable on;                       
    app_protect_policy_file /<path-to-config-dir-on-the-instance>/policy-name.json; 
    app_protect_security_log_enable on;
    app_protect_security_log /<path-to-config-dir-on-the-instance>/log-profile-name.json; 

NGINX App Protect policy deployment status not reflecting removal of associated instance.

Issue ID Status
38700 Open

Description

This error occurs when you use NGINX App Protect on the Instance Manager and then remove all NGINX App Protect references from the NGINX Config. As a result, it incorrectly shows some NAP policies as still being deployed on this instance.

Workaround

  1. Login to the instance sudo rm /etc/nms/app_protect_metadata.json

  2. Restart nginx-agent sudo systemctl restart nginx-agent


Unreferenced NGINX App Protect policy file in /etc/nms

Issue ID Status
38488 Open

Description

When using NGINX Instance Manager with App Protect policies, previously referenced policies in the NGINX configuration may not be removed after they are no longer referenced in the NGINX config.

Workaround

Unreferenced policy files may be removed manually from /etc/nms.


App Protect UI fails when deployed via Helm chart

Issue ID Status
38782 Open

Description

When installing NGINX Instance Manager on Kubernetes via Helm Chart, the App Protect page shows an error banner, and no default policies are displayed.


External references are not supported in policies.

Issue ID Status
36265 Open

Description

References to external files in a policy are not supported.

For example, in the NGINX App Protect WAF JSON declarative policy, these references are not supported:

  • User-defined signatures
  • Security controls in external references
  • Referenced OpenAPI Spec files

After upgrading to Instance Manager 2.6.0, the data plane manager service crashes if you publish an NGINX configuration with the NGINX App Protect enablement directive set to ON

Issue ID Status
38904 Open

Description

After upgrading Instance Manager to v2.6.0 from v2.5.0 or v2.5.1, the data plane manager (dpm) service might crash. This happens when you publish an NGINX configuration with the NGINX App Protect enablement directive (app_protect_enable) set to ON. Also, Instance Manager does not return updated status information for NGINX App Protect instances.


When upgrading a multi-node NMS deployment with helm charts the ingestion pod may report a “Mismatched migration version” error

Issue ID Status
38880 Open

Description

When using the Instance Manager Helm upgrade command on a multi worker node kubernetes cluster setup, the ingestion pod may report a “Mismatched migration version” error.

Workaround

Post upgrade, do the following steps:

kubectl -n nms scale --replicas=0 deployment.apps/dpm; kubectl -n nms scale --replicas=1 deployment.apps/dpm
kubectl -n nms scale --replicas=0 deployment.apps/ingestion; kubectl -n nms scale --replicas=1 deployment.apps/ingestion

When upgrading Instance Manager from v2.4 to later versions of Instance Manager, certificate associations are no longer visible.

Issue ID Status
38641 Open

Description

When upgrading Instance Manager from 2.4 or earlier to 2.5 or later, certificates associated with an instance will no longer reflect the association. This does not impact the functioning of the data plane instance or usage of the certificate and key.

Workaround

Re-associate any certificates that no longer reflect the association.


Config deployment could fail when referencing remote cert inside allowed directories

Issue ID Status
38596 Open

Description

Deploying NGINX config with references to remote cert that resides in allowed directories could fail, with the following error:

BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory.

This can also be diagnosed with log entries in /var/log/nginx-agent/agent.log, noting the removal of the referenced certificate.

Workaround

  • Add the referenced cert to NMS as managed certificate and publish the config again.
  • Move the referenced remote certificate to a directory that’s not in the allowed directory list.

When upgrading a multi-node NMS deployment with helm charts the core, dpm, or integrations pods may fail to start

Issue ID Status
38589 Open

Description

When using the NMS Instance Manager Helm upgrade command on a multi worker node kubernetes cluster setup, the core, dpm and integrations deployments may fail to upgrade.

Workaround

Post upgrade, do the following steps:

kubectl -n nms scale --replicas=0 deployment.apps/dpm; kubectl -n nms scale --replicas=1 deployment.apps/dpm
kubectl -n nms scale --replicas=0 deployment.apps/core; kubectl -n nms scale --replicas=1 deployment.apps/core
kubectl -n nms scale --replicas=0 deployment.apps/integrations; kubectl -n nms scale --replicas=1 deployment.apps/integrations

Null data count is not correctly represented in the NGINX Plus usage graph.

Issue ID Status
38206 Open

Description

When there is no NGINX Plus instance reporting to NGINX Management Suite, but there was some in the past, the graph in the UI assumes the last reported value for those times. The GUI and API both report the Min as the lowest reported number and do not factor in periods where no instances were reported. Similarly, the average value is calculated only using reported values and ignores time periods where no instances are reported.


HTTP version schema returns incorrect value in Advanced metrics module

Issue ID Status
38041 Open

Description

The values currently populated for http.version_schema are incorrect. The response is “4” for HTTP traffic and “6” for HTTPS traffic.


Count of NGINX Plus graph has a delay in being populated

Issue ID Status
37705 Open

Description

When viewing the NGINX Plus usage in Instance Manager, the graph displaying usage over time requires several hours of data before displaying the count.

Workaround

The data presented in the graph can be retrieved from the API.


2.5.0

Instance Manager reports the NGINX App Protect WAF build number as the version

Issue ID Status
37510 Open

Description

After installing NGINX App Protect WAF 3.12 on the data plane and enabling NAP status reporting for the NGINX Agent, Instance Manager reports the App Protect version as build-3.1088.1 instead of 3.12.

This version mismatch does not affect system operations.


Aux data fails to upload if the size is greater than 3145728 characters

Issue ID Status
37498 Open

Description

Updating a config with an aux data file exceeding 3145728 characters fails with a validation error similar to the following example:

Request body has an error: doesn’t match the schema: Error at “/auxFiles/files/3/contents”: maximum string length is 3145728


Staged configs fail to publish after upgrading NGINX Management Suite

Issue ID Status
37479 Open

Description

After upgrading NGINX Management Suite to 2.5.0, when you try to publish a staged config from the web interface, the system returns an error similar to the following:

“The published configuration is older than the active instance configuration.”

Workaround

Make a minor edit to a staged config, such as adding a space, then save the change. You should be able to publish now.


“Deployment Not Found” error when publishing NGINX config to NATS server

Issue ID Status
37437 Open

Description

Occasionally, when publishing an NGINX config to a NATS server, the system returns a Deployment Not Found error, and the nms.log file includes the following error:

http failure with code ‘131043’: <nil>.

Workaround

Remove the existing NATs working directory and restart the NMS Data Plane Manager (nms-dpm) service as root.

Caution:
Restarting the nms-dpm service is disruptive and may result in the loss of event data. You should schedule a maintenance window for restarting the service.
rm -rf /var/lib/nms/streaming
systemctl restart nms-dpm

Instance Manager returns a “Download failed” error when editing an NGINX config for instances compiled and installed from source

Issue ID Status
35851 Open

Description

When an NGINX instance is compiled and installed from source without a prefix, if you edit the instance’s config in Instance Manager, the system reports an error similar to the following example:

Error getting the NGINX instance configuration: failed to download the configuration. Check /var/log/nms/nms.log and nginx-agent logs for errors.

Workaround

To resolve or avoid this issue, take the following steps to compile and install NGINX from source with a prefix and restart the NGINX Agent:

  1. Stop the NGINX instance:

    sudo stop nginx
    
  2. Locate the source code for the NGINX instance.

  3. Add a prefix:

    ./auto/configure --prefix=</path/to/nginx> --without-http_rewrite_module
    
  4. Compile and install NGINX:

    sudo make install
    
  5. Start NGINX:

    cd </path/to/nginx/sbin/>
    ./nginx
    
  6. Restart the NGINX Agent:

    service nginx-agent restart
    

2.4.0

Extended NGINX metrics aren’t reported for NGINX Plus R26 and earlier

Issue ID Status
37738 Fixed in 2.5.1

Description

The NGINX Agent installed with Instance Manager 2.4.0 and 2.5.0 uses the wrong API version for NGINX Plus R26 and earlier. As a result, extended metrics are not reported, and an error similar to the following is logged:

“Failed to create plus metrics client: API version of the client is not supported by running NGINX Plus.”


Instance Scan overview page doesn’t scroll to show the full list of instances

Issue ID Status
36514 Fixed in 2.5.0

Description

In the Instance Manager web interface, the Instance Scan overview page (Instance Manager > Scan), which lists the identified NGINX instances, does not scroll.

Workaround

You may need to zoom out of the page using your browser’s settings to view a long list of instances.


Publishing an instance group config with an aux file outside the allowed directory fails

Issue ID Status
36410 Open

Description

If an instance group’s configuration references an aux file (for example, an SSL certificate) that is not in the expected allowed directory, publishing the config will fail. The system returns an error similar to the following:

Config apply failed (write): the file <filename> is outside the allowed directory list.

Workaround

Move the aux file to the allowed directory and update the configuration; for example, use /etc/nginx/ for certificates.


Managed certificates may be overwritten if they have the same name on different data path certificates

Issue ID Status
36240 Fixed in 2.5.0

Description

When synchronizing data path configurations, the NGINX Agent may overwrite a managed certificate on NGINX Management Suite if multiple data paths have managed certificates with the same name.

Workaround

Use unique names for each certificate.


Installing NGINX Agent on Ubuntu 22.04 LTS fails with 404 Not Found error

Issue ID Status
35339 Open

Description

When installing the NGINX Agent on Ubuntu 22.04 LTS, the installation script fails with a 404 Not Found error similar to the following:

404 Not found [IP: <IP address>]
Reading package lists...
E: The repository 'https://192.0.2.0/packages-repository/deb/ubuntu jammy Release' does not have a Release file.
E: The repository 'https://pkgs.nginx.com/app-protect/ubuntu jammy Release' does not have a Release file.
E: The repository 'https://pkgs.nginx.com/app-protect-security-updates/ubuntu jammy Release' does not have a Release file.

Workaround

Edit the NGINX Agent install script to use the codename focal for Ubuntu 20.04.

  1. Download the installation script:

    curl -k https://<NGINX-INSTANCE-MANAGER-FQDN>/install/nginx-agent > install.sh
    
  2. Open the install.sh file for editing.

  3. Make the following changes:

    On lines 256-258, change the following:

    codename=$(cat /etc/*-release | grep '^DISTRIB_CODENAME' | 
    sed 's/^[^=]*=\([^=]*\)/\1/' | 
    tr '[:upper:]' '[:lower:]')
    

    to:

    codename=focal
    

    —OR—

    Alternatively, on line 454, change the following:

    deb ${PACKAGES_URL}/deb/${os}/ ${codename} agent
    

    to:

    deb ${PACKAGES_URL}/deb/${os}/ focal agent
    
  4. Save the changes.

  5. Run the install.sh script.


2.3.0

Unable to publish config changes to a custom nginx.conf location

Issue ID Status
35276 Open

Description

Using Instance Manager to publish a config may fail if the NGINX Instance has the configuration file stored in a folder other than the default.

Workaround

Use the default location for the configuration file.


Scan misidentifies some NGINX OSS instances as NGINX Plus

Issue ID Status
35172 Open

Description

When NGINX Plus is installed on a data path instance, then removed and replaced with NGINX OSS, Instance Manager may incorrectly identify the instance as an NGINX Plus instance. This is due to multiple NGINX entries for the same data path.

Workaround

Use Instance Manager’s NGINX Instances API to remove the inactive NGINX instance. For instructions, refer to the API reference guide, which you can find at https://<NGINX-INSTANCE-MANAGER-FQDN>/ui/docs.

You may need to stop the NGINX Agent first. To stop the NGINX Agent, run the following command:

sudo systemctl stop nginx-agent

Metrics may report additional data

Issue ID Status
34255 Open

Description

Instance Manager reports metrics at a per-minute interval and includes dimensions for describing the metric data’s characteristics.

An issue has been identified in which metric data is aggregated across all dimensions, not just for existing metrics data. When querying the Metrics API with aggregations like SUM(metric-name), the aggregated data causes the API to over count the metric. This overcounting skews some of the metrics dashboards.

Workaround

When querying the Metrics API, you can exclude the data for an aggregated dimension by specifying the dimension name in the filterBy query parameter.

filterBy=<dimension-name>!= ''

2.2.0

Post-install steps to load SELinux policy are in the wrong order

Issue ID Status
34276 Fixed in 2.3.0

Description

After installing Instance Manager, the post-installation steps shown in the command-line output say to start the NGINX Management Suite services before loading the included SELinux policy. These steps are transposed. You need to load the SELinux policy first and start the NMS services after; otherwise, the NMS services will be unconfined.

Workaround

See the Enforce Security with SELinux Policy guide for the correct instructions.


Giving long names (255+ characters) to certificates causes internal error

Issue ID Status
34185 Open

Description

When adding certificates, an internal error (error code: 134018) is returned if the name given for the certificate exceeds 255 characters.

Workaround

Use a name that is 255 or fewer characters.


Associating instances with expired certificates causes internal error

Issue ID Status
34182 Open

Description

An internal error (error code: 134018) is reported when adding an expired certificate if the certificate is associated with an instance.

Workaround

If possible, use a certificate that isn’t expired.


2.1.0

Duplicate instances are shown after upgrading Agent to 2.1.0

Issue ID Status
33307 Open

Description

After upgrading to Instance Manager 2.1.0, and updating nginx-agent from platform packaging, duplicate instances may appear on the Instance overview page. This issue is caused by a change in how the NGINX Agent generates the system_uid.

Workaround

You can safely delete the older entries or wait for them to expire.


Helm chart deployments don’t support OIDC

Issue ID Status
33248 Open

Description

OIDC is not supported for helm chart deployments of Instance Manager on Kubernetes. Only basic auth is supported.


“No such process” error occurs when publishing a configuration

Issue ID Status
33160 Open

Description

When publishing a configuration, you might encounter an error similar to the following example:

config action failed: Config apply failed (write): no such process

This error can occur when there is a desyncronization between the NGINX Agent and NGINX PID, often after manually restarting NGINX when the Agent is running.

Workaround

Restart the NGINX Agent:

sudo systemctl restart nginx-agent

2.0.0

“Option unknown” password error when installing Instance Manager on Ubuntu with OpenSSL v1.1.0

Issue ID Status
33055 Fixed in 2.6.0

Description

When installing Instance Manager on Ubuntu with OpenSSL v1.1.0, you might see an error similar to the following example:

  Generating default password for ‘admin’ user account
  Using openssl version 1.1
  passwd: Option unknown option -6
  passwd: Use -help for summary.
  dpkg: error processing package nms-instance-manager

If you receive this error during the installation, your version of OpenSSL is not compatible with the password hashing algorithm used to generate an initial admin password.

Workaround

Upgrade your OpenSSL version to v1.1.1 or later:

  sudo apt-get install openssl

After updating OpenSSL this way, apt will re-try installing Instance Manager.


NGINX App Protect WAF blocks Instance Manager from publishing configurations

Issue ID Status
32718 Open

Description

Instance Manager does not currently support managing NGINX App Protect WAF instances. NGINX App Protect WAF may block attempts to publish configurations to NGINX App Protect WAF instances.


Instance Manager reports old NGINX version after upgrade

Issue ID Status
31225 Open

Description

After upgrading NGINX to a new version, the Instance Manager web interface and API report the old NGINX version until the NGINX Agent is restarted.

Workaround

Restart the Agent to have the new version reflected properly:

systemctl restart nginx-agent

Web interface doesn’t report error when failing to upload large config files

Issue ID Status
31081 Open

Description

In the web interface, when uploading a config file that’s larger than 50 MB (max size), the system incorrectly reports the state as Analyzing (Status code 403), although the upload failed.

Workaround

Keep config files under 50 MB.


CentOS 7, RHEL 7, and Amazon Linux 2 package managers allow unsupported NGINX/NGINX Plus versions

Issue ID Status
28758 Open

Description

When installing on CentOS 7, RHEL 7, and Amazon Linux 2, the package manager doesn’t prevent installing Instance Manager with unsupported versions of NGINX or NGINX Plus. As a consequence, it is possible that nms-instance-manager is installed without an NGINX gateway. Resulting in a less than optimal experience.

Workaround

Install a supported version of NGINX (v1.18 or later) or NGINX Plus (R22 or later). See the Technical Specifications guide for details.


gRPC errors when starting Instance Manager

Issue ID Status
28683 Open

Description

When starting Instance Manager, you may see errors similar to the following in /etc/nginx/conf.d/nms-http.conf:227:

nginx[1234]: nginx: [emerg] unknown directive "grpc_socket_keepalive"

Workaround

Make sure your version of NGINX is v1.18 or later. See the Technical Specifications guide for details.