NGINX App Protect DoS Troubleshooting Guide

Overview

This Troubleshooting Guide is intended to provide guidance to customers in the detection and correction of programming issues in F5 NGINX App Protect DoS. It may also be useful to IT.

Resolving Known Problems

Configuration

Problem	Solution
NGINX is not running (ps -aux) Reloading NGINX fails	Check the error log at `/var/log/nginx/error.log`. Fix the problem and re-run NGINX.
No original source IP in logs	1. XFF is not configured (or not configured correctly) 2. External Load Balancer doesn’t forward XFF
NGINX App Protect DoS functionality is not as expected	NGINX App Protect DoS has several logs which can be used for troubleshooting. Usually, it is best to look for any warning or error messages within the logs. Refer to Logs Overview
`Too many open files` error message	Increase number of file descriptors. For example: `worker_rlimit_nofile 65535;` in the main context of `nginx.conf` file. Refer to worker_rlimit_nofile directive
`setrlimit ... failed (Permission denied)` error message	Increase the limit using the following command as the root user: `setsebool -P httpd_setrlimit 1;` Refer to Issue 4: Too many files are open Error
More protected objects than expected	The `app_protect_dos_enable` directive is inherited by all server and location blocks beneath it, each block will be a protected object. Consider moving this directive from outer to inner block. Refer to: NGINX App Protect DoS - Directives and Policy
`No DOS protection for ngx_worker at idx X` warning message	There are more nginx processes than allowed. Either decrease the number of nginx processes (ngx_processes directive in `nginx.conf` file) or increase the number of supported workers for NGINX App Protect DoS using the flag `--max-workers NUM` for `/usr/bin/adminstall`.
`unknown directive 'app_protect_dos_xxx'` error message	App Protect DOS module is not loaded. Add this line to the main (global) context of nginx.conf: `load_module "/etc/nginx/modules/ngx_http_app_protect_dos_module.so";`
NGINX struggles handling a high rate of incoming connections	Linux machine should be tuned for optimal performance. Refer to Tuning NGINX for Performance
Error in `adminstall` process, such as `Failed to allocate`	Insufficient memory to allocate all the required resources. Increase the `--memory` size or decrease the number of nginx workers (`--max_workers`) if not all of them are going to be in use. Use the `--help` flag for more info.

ELK issues

ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for NGINX App Protect DoS GitHub repo.

SELinux

Configure SELinux to allow NGINX App Protect DoS.

The configuration steps are found in the SELinux configuration section of the deployment guide.

If SELinux still denies access to something, it means that one of more security exceptions should be enabled.

The following steps describe how to find the problematic exception and enable it.

Temporarily add the httpd_t domain to the permissive list (this will completely enable all the fields).
In this configuration SELinux will not deny anything related to NGINX as NGINX is labeled with the httpd_t context.

semanage permissive -a httpd_t

Repeat the scenario which made SELinux deny and see that it now works.
In permissive mode, security exceptions are logged to the default Linux audit log. Due to the previous step, the permitted exception will be logged.
The log can be found in /var/log/audit/audit.log.
The following command will parse the audit log and build a SELinux command that will permit all the exceptions found in the log:

grep nginx /var/log/audit/audit.log | audit2allow -m nginx

Compare the generated output to the nginx.te file mentioned in the deployment guide. Add all the missing commands to the nginx.te file and repeat the SELinux configuration mentioned in the deployment guide.
Delete the httpd_t domain from the permissive list:

semanage permissive -d httpd_t

For more information about how to use NGINX Plus with SELinux - check our blog

Send Logs to Support

If there are any problems, collect the troubleshooting information in a tarball and send it to your customer support engineer.

Get package version:

a. Get NGINX App Protect DoS version:

/usr/bin/admd -v > package_versions.txt

b. Get packages version:
For CentOS/RHEL:

rpm -qa nginx-plus* app-protect* >> package_versions.txt

For Debian/Ubuntu:

apt list --installed | grep -E 'nginx-plus|app-protect' >> package_versions.txt

c. Get OS version:

cat /etc/os-release > system_version.txt && uname -r >> system_version.txt && cat /proc/version >> system_version.txt

d. Get NGINX App Protect DoS shared memory dump:

admd -c > napd_shmem.txt

e. Get Linux shared memory dump:

ipcs -m > linux_shmem.txt

Create a list of files for tarball:
a. Create a file using your favorite editor (i.e VI editor)
```
vi logs.txt
```
b. Insert the following content into the file created above:
```
package_versions.txt
system_version.txt
napd_shmem.txt
linux_shmem.txt
/var/log/adm/*
/var/run/adm/*
/var/log/nginx/*
```
c. Add the path of your NGINX configuration files including all references, for example:
```
/etc/nginx/nginx.conf
/etc/nginx/conf.d/*
```
d. Add all policies and log file configuration, for example:
```
/etc/app_protect_dos/*
```
Create the tarball:
```
tar cvfz logs.tgz `cat logs.txt`
```
Send logs.tgz to your customer support.

Last modified August 22, 2024