End of Sale Notice:

F5 NGINX is announcing the End of Sale (EoS) for NGINX Controller API Management Module, effective January 1, 2024.

F5 maintains generous lifecycle policies that allow customers to continue support and receive product updates. Existing NGINX Controller API- Management customers can continue to use the product past the EoS date. License renewals are not available after September 30, 2024.

See our End of Sale announcement for more details.

End of Sale Notice:

F5 NGINX is announcing the End of Sale (EoS) for NGINX Controller Application Delivery Module, effective January 1, 2024.

F5 maintains generous lifecycle policies that allow customers to continue support and receive product updates. Existing NGINX Controller Application Delivery customers can continue to use the product past the EoS date. License renewals are not available after September 30, 2024.

See our End of Sale announcement for more details.

Troubleshoot NGINX Controller and the Controller Agent

Steps to take to investigate and fix issues with NGINX Controller and the Controller Agent.

Overview

If NGINX isn’t behaving how you expect, you can take the following steps to troubleshoot issues. If you need to contact NGINX Support, make sure to create a support package first.

Fix NGINX Controller Issues by Upgrading

We recommend you upgrade NGINX Controller as new versions become available. Upgrades include new features, feature improvements, or fixes for known issues.

To look up your version of NGINX Controller:

Open the NGINX Controller browser interface and log in.
Select the NGINX Controller menu icon, then select Platform.
On the Platform menu, select Cluster > Overview.

See Also:
Refer to the NGINX Controller release notes to see what’s new in the latest release of NGINX Controller.

Create a Support Package

You can create a support package for NGINX Controller that you can use to diagnose issues.

Note:
You will need to provide a support package if you open a ticket with NGINX Support via the MyF5 Customer Portal.

/opt/nginx-controller/helper.sh supportpkg [-o|--output <file name>] [-s|--skip-db-dump] [-t|--timeseries-dump <hours>]

Options	Description
`-o` \| `--output`	Save the support package file to `<file name>`.
`-s` \| `--skip-db-dump`	Don’t include the database dump in the support package.
`-t` \| `--timeseries-dump <hours>`	Include the last `<n hours>` of timeseries data in the support package (default 12 hours).

Take the following steps to create a support package:

Open a secure shell (SSH) connection to the NGINX Controller host and log in as an administrator.
Run the helper.sh utility with the supportpkg option:
```
/opt/nginx-controller/helper.sh supportpkg
```
The support package is saved to:

/var/tmp/supportpkg-<timestamp>.tar.gz

For example:

/var/tmp/supportpkg-20200127T063000PST.tar.gz
Run the following command on the machine where you want to download the support package to:
```
scp <username>@<controller-host-ip>:/var/tmp/supportpkg-<timestamp>.tar.gz /local/path
```

Support Package Details

The support package is a tarball that includes NGINX Controller configuration information, logs, and system command output. Sensitive information, including certificate keys, is not included in the support package.

The support package gathers information from the following locations:

.
├── database
│   ├── common.dump                                - full dump of the common database
│   ├── common.dump_stderr                         - any errors when dumping the database
│   ├── common-apimgmt-api-client-api-keys.txt     - contents of apimgmt_api_client_api_keys table from the common database
│   ├── common-apimgmt-api-client-groups.txt       - contents of apimgmt_api_client_groups table from the common database
│   ├── common-email-verification.txt              - contents of email_verification table from the common database
│   ├── common-oauth-clients.txt                   - contents of oauth_clients table from the common database
│   ├── common-settings-license.txt                - contents of settings_license table from the common database
│   ├── common-settings-nginx-plus.txt             - contents of settings_nginx_plus table from the common database
│   ├── common-table-size.txt                      - list of all tables and their size in the common database
│   ├── data-table-size.txt                        - list of all tables and their size in the data database
│   ├── postgres-database-size.txt                 - size of every database
│   ├── postgres-long-running-queries.txt          - all queries running longer than 10 seconds
│   ├── system.dump                                - full dump of the system database
│   ├── system-account-limits.txt                  - contents of account_limits table from the system database
│   ├── system-accounts.txt                        - contents of accounts table from the system database
│   ├── system-deleted-accounts.txt                - contents of deleted_accounts table from the system database
│   ├── system-deleted-users.txt                   - contents of deleted_users table from the system database
│   ├── system-users.txt                           - contents of users table from the system database
│   └── system-table-size.txt                      - list of all tables and their size in the system database
├── k8s                                            - output of `kubectl cluster-info dump -o yaml` augmented with some extra info
│   ├── apiservices.txt                            - output of `kubectl get apiservice`
│   ├── kube-system                                - contents of the kube-system namespace
│   │   ├── coredns-5c98db65d4-6flb9
│   │   │   ├── desc.txt                           - pod description
│   │   │   ├── logs.txt                           - current logs
│   │   │   └── previous-logs.txt                  - previous logs, if any
│   │   ├── ...
│   │   ├── daemonsets.yaml                        - list of daemonsets
│   │   ├── deployments.yaml                       - list of deployments
│   │   ├── events.yaml                            - all events in this namespace
│   │   ├── namespace.yaml                         - details of the namespace, including finalizers
│   │   ├── pods.txt                               - output of `kubectl get pods --show-kind=true -o wide`
│   │   ├── pods.yaml                              - list of all pods
│   │   ├── replicasets.yaml                       - list of replicasets
│   │   ├── replication-controllers.yaml           - list of replication controllers
│   │   ├── resources.txt                          - all Kubernetes resources in this namespace
│   │   └── services.yaml                          - list of services
│   ├── nginx-controller                           - contents of the nginx-controller namespace
│   │   ├── apigw-8fb64f768-9qwcm
│   │   │   ├── desc.txt                           - pod description
│   │   │   ├── logs.txt                           - current logs
│   │   │   └── previous-logs.txt                  - previous logs, if any
│   │   ├── ...
│   │   ├── daemonsets.yaml                        - list of daemonsets
│   │   ├── deployments.yaml                       - list of deployments
│   │   ├── events.yaml                            - all events in this namespace
│   │   ├── namespace.yaml                         - details of the namespace, including finalizers
│   │   ├── pods.txt                               - output of `kubectl get pods --show-kind=true -o wide`
│   │   ├── pods.yaml                              - list of all pods
│   │   ├── replicasets.yaml                       - list of replicasets
│   │   ├── replication-controllers.yaml           - list of replication controllers
│   │   ├── resources.txt                          - all Kubernetes resources in this namespace
│   │   ├── services.yaml                          - list of services
│   ├── nodes.txt                                  - output of `kubectl describe nodes`
│   ├── nodes.yaml                                 - list of nodes
│   ├── resources.txt                              - all non-namespaced Kubernetes resources (including PersistentVolumes)
│   └── version.yaml                               - Kubernetes version
├── logs                                           - copy of /var/log/nginx-controller/
│   └── nginx-controller-install.log
├── os
│   ├── cpuinfo.txt                                - output of `cat /proc/cpuinfo`
│   ├── df-h.txt                                   - output of `df -h`
│   ├── df-i.txt                                   - output of `df -i`
│   ├── docker-container-ps.txt                    - output of `docker container ps`
│   ├── docker-images.txt                          - output of `docker images`
│   ├── docker-info.txt                            - output of `docker info`
│   ├── docker-stats.txt                           - output of `docker stats --all --no-stream`
│   ├── docker-version.txt                         - output of `docker version`
│   ├── du-mcs.txt                                 - output of `du -mcs /opt/nginx-controller/* /var/log /var/lib`
│   ├── env.txt                                    - output of `env`
│   ├── firewall-cmd.txt                           - output of `firewall-cmd --list-all`
│   ├── free.txt                                   - output of `free -m`
│   ├── hostname-all-fqdns.txt                     - output of `hostname --all-fqdns`
│   ├── hostname-fqdn.txt                          - output of `hostname --fqdn`
│   ├── hostname.txt                               - output of `hostname`
│   ├── hostsfile.txt                              - output of `cat /etc/hosts`
│   ├── ip-address.txt                             - output of `ip address`
│   ├── ip-neigh.txt                               - output of `ip neigh`
│   ├── ip-route.txt                               - output of `ip route`
│   ├── iptables-filter.txt                        - output of `iptables -L -n -v`
│   ├── iptables-mangle.txt                        - output of `iptables -L -n -v -t mangle`
│   ├── iptables-nat.txt                           - output of `iptables -L -n -v -t nat`
│   ├── iptables-save.txt                          - output of `iptables-save`
│   ├── journal-kubelet.txt                        - output of `journalctl -q -u kubelet --no-pager`
│   ├── lspci.txt                                  - output of `lspci -vvv`
│   ├── netstat-nr.txt                             - output of `netstat -nr`
│   ├── ps-faux.txt                                - output of `ps faux`
│   ├── pstree.txt                                 - output of `pstree`
│   ├── ps.txt                                     - output of `ps aux --sort=-%mem`
│   ├── resolvconf.txt                             - output of `cat /etc/resolv.conf`
│   ├── selinux-mode.txt                           - output of `getenforce`
│   ├── ss-ltunp.txt                               - output of `ss -ltunp`
│   ├── swapon.txt                                 - output of `swapon -s`
│   ├── sysctl.txt                                 - output of `sysctl -a --ignore`
│   ├── systemd.txt                                - output of `journalctl -q --utc`
│   ├── top.txt                                    - output of `top -b -o +%CPU -n 3 -d 1 -w512 -c`
│   ├── uname.txt                                  - output of `uname -a`
│   ├── uptime.txt                                 - output of `cat /proc/uptime`
│   └── vmstat.txt                                 - output of `cat /proc/vmstat`
├── timeseries
│   ├── table-sizes.stat                           - stat table containing controller table sizes
│   ├── events.csv                                 - events table dump in csv
│   ├── events.sql                                 - events table schema
│   ├── metrics_1day.csv                           - metrics_1day table dump in csv
│   ├── metrics_1day.sql                           - metrics_1day table schema
│   ├── metrics_1hour.csv                          - metrics_1hour table dump in csv
│   ├── metrics_1hour.sql                          - metrics_1hour table schema
│   ├── metrics_5min.csv                           - metrics_5min table dump in csv
│   ├── metrics_5min.sql                           - metrics_5min table schema
│   ├── metrics.csv                                - metrics table dump in csv
│   ├── metrics.sql                                - metrics table schema
│   ├── system-asynchronous-metrics.stat           - shows info about currently executing events or consuming resources
│   ├── system-events.stat                         - information about the number of events that have occurred in the system
│   ├── system-metrics.stat                        - system metrics
│   ├── system-parts.stat                          - information about parts of a table in the MergeTree family
│   ├── system-settings.stat                       - information about settings that are currently in use
│   └── system-tables.stat                         - information about all the tables
└── version.txt                                    - Controller version information

Security Events Not Available

If NGINX Controller isn’t logging WAF Violation Security Events for an App Component that has WAF enabled, take the following steps:

Check the agent.conf security setting for every Instance referenced by the Gateway(s) associated with the App Component. You’ll need to verify that the Extensions group contains the setting security = True.
Restart the NGINX Controller Agent.

To start, stop, and restart the NGINX Controller Agent, run the following commands on the NGINX Plus system where you installed the Agent.

Start the NGINX Controller Agent:

service controller-agent start

Stop the NGINX Controller Agent:

service controller-agent stop

Restart the NGINX Controller Agent:

service controller-agent restart

Signature Name missing in Security Violation Events

If you don’t see Signature Names in Security Violation Events, restart the Controller Agent on the dataplane instance.

sudo systemctl restart controller-agent

Controller Agent Install Script Failed to Download

When deploying an NGINX Plus instance, the deployment may fail because the Controller Agent install script doesn’t download. When this happens, an error similar to the following is logged to /var/log/agent_install.log: “Failed to download the install script for the agent.”

Take the following steps to troubleshoot the issue:

Ensure that ports 443 and 8443 are open between NGINX Controller and the network where the NGINX Plus instance is being deployed.
Verify that you can communicate with NGINX Controller from the NGINX Plus instance using the NGINX Controller FQDN that you provided when you installed NGINX Controller.
If you’re deploying an NGINX Plus instance on Amazon Web Services using a template, ensure that the Amazon Machine Image (AMI) referenced in the instance_template has a cURL version of 7.32 or newer.

Controller Agent Asks for Password

If the system asks you to provide a password when you’re installing the Controller Agent, the cause may be that you are starting the install script from a non-root account. If so, you need sudo rights. Depending on your system configuration, sudo may ask you for a password if you’re using a non-root account.

Controller Agent Isn’t Reporting Metrics

After you install and start the Controller Agent, it should begin reporting right away, pushing aggregated data to NGINX Controller at regular one-minute intervals. It takes about one minute for a new Instance to appear in the NGINX Controller user interface.

If you don’t see the new Instance in the user interface or the Controller Agent isn’t collecting metrics, make sure of the following:

The Controller Agent package – nginx-controller-agent – installed successfully without any warnings.
The controller-agent service is running and updating its log file. To check the status, run the following command:
```
systemctl status controller-agent
```
See Also:
For troubleshooting purposes, you can turn on Controller Agent debug logging by editing the agent.conf file. For more information, refer to K64001240: Enabling NGINX Controller Agent debug logging.
The system DNS resolver is correctly configured, and the NGINX Controller server’s fully qualified domain name (FQDN) can be resolved.
The controller-agent service can be running as root or a different user, chosen during the installation if the Controller Agent was installed to run as a non-root user. To view the user ID for the controller-agent service, run the following command:
```
ps -ef | egrep 'agent'
```
The output looks similar to the following (with a different user for non-root Agent installations):
```
root     19132     1  1 Sep03 ?        00:23:45 /usr/bin/nginx-controller-agent
```
The Controller Agent and the NGINX Instance user IDs can both run the ps command to see all the system processes. If the ps command is restricted for non-privileged users, the Controller Agent won’t detect the NGINX master process.
The system time is set correctly. If the time on the system where the Controller Agent is running is ahead or behind the NGINX Controller’s system time, you won’t be able to see data in graphs. Make sure that NGINX Controller and any NGINX Instances have their time synchronized using NTP.
The NGINX Plus API is set up correctly and working.

Refer to the Configuring the API section of the NGINX Plus Admin Guide for instructions.
All NGINX configuration files are readable by the Controller Agent user ID. Verify the owner, group, and permissions settings.
selinux, apparmor, or other third-party OS security tools are not interfering with the metrics collection. For example, for selinux, inspect /etc/selinux/config and try setenforce 0 temporarily to see if it improves the situation for certain metrics.
The virtual private server (VPS) provider has not used hardened Linux kernels that may restrict non-root users from accessing /proc and /sys. Metrics describing the system and NGINX disk I/O are usually affected. There is no easy workaround for this except to allow the Controller Agent to run as root. Fixing permissions for /proc and /sys/block may also help.

See Also:

For more information on installing and configuring the Controller Agent, see the following topics:

Installing the NGINX Controller Agent

Installing the NGINX Controller Agent for non-root users

Configuring the NGINX Controller Agent

Configuring metrics collection for NGINX Controller

Licensing

If NGINX Controller appears to be unlicensed after a version upgrade, try the following options to resolve the issue.

Events Don’t Display in the Analytics User Interface

Certain content-filtering and ad-blocking web browser extensions may incorrectly block the elements on the NGINX Controller Analytics events page. As a result, when you access the Analytics > Events page using the NGINX Controller user interface, you may observe messages indicating missing events. Refer to the AskF5 KB article K48603454 to learn more about this issue and how to resolve it.