NGINX App Protect DoS Troubleshooting Guide

Learn about the NGINX App Protect DoS Troubleshooting Guide

Overview

This Troubleshooting Guide is intended to provide guidance to customers in the detection and correction of programming issues in NGINX App Protect DoS. It may also be useful to IT.

Resolving Known Problems

Configuration

Problem Solution
NGINX is not running (ps -aux)

Reloading NGINX fails
Check the error log at /var/log/nginx/error.log.
Fix the problem and re-run NGINX.
No original source IP in logs 1. XFF is not configured (or not configured correctly)
2. External Load Balancer doesn’t forward XFF
NGINX App Protect DoS functionality is not as expected NGINX App Protect DoS has several logs which can be used for troubleshooting.
Usually, it is best to look for any warning or error messages within the logs.
Refer to Logs Overview
Too many open files error message Increase number of file descriptors.
For example: worker_rlimit_nofile 65535; in the main context of nginx.conf file.
Refer to worker_rlimit_nofile directive
More protected objects than expected The app_protect_dos_enable directive is inherited by all server and location blocks beneath it, each block will be a protected object.
Consider moving this directive from outer to inner block.
Refer to: NGINX App Protect DoS - Directives and Policy
No DOS protection for ngx_worker at idx X warning message There are more nginx processes than allowed.
Either decrease the number of nginx processes (ngx_processes directive in nginx.conf file) or increase the number of supported workers for NGINX App Protect DoS using the flag --max-workers NUM for /usr/bin/adminstall.
unknown directive 'app_protect_dos_xxx' error message App Protect DOS module is not loaded. Add this line to the main (global) context of nginx.conf:
load_module "/etc/nginx/modules/ngx_http_app_protect_dos_module.so";
NGINX struggles handling a high rate of incoming connections Linux machine should be tuned for optimal performance.
Refer to Tuning NGINX for Performance

ELK issues

ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for F5 NGINX App Protect DoS GitHub repo .

SELinux

Configure SELinux to allow NGINX App Protect DoS.

The configuration steps are found in the SELinux configuration section of the deployment guide.

If SELinux still denies access to something, it means that one of more security exceptions should be enabled.

The following steps describe how to find the problematic exception and enable it.

  1. Temporarily add the httpd_t domain to the permissive list (this will completely enable all the fields).
    In this configuration SELinux will not deny anything related to NGINX as NGINX is labeled with the httpd_t context.
semanage permissive -a httpd_t
  1. Repeat the scenario which made SELinux deny and see that it now works.
  2. In permissive mode, security exceptions are logged to the default Linux audit log. Due to the previous step, the permitted exception will be logged.
    The log can be found in /var/log/audit/audit.log.
  3. The following command will parse the audit log and build a SELinux command that will permit all the exceptions found in the log:
grep nginx /var/log/audit/audit.log | audit2allow -m nginx
  1. Compare the generated output to the nginx.te file mentioned in the deployment guide. Add all the missing commands to the nginx.te file and repeat the SELinux configuration mentioned in the deployment guide.
  2. Delete the httpd_t domain from the permissive list:
semanage permissive -d httpd_t

For more information about how to use NGINX Plus with SELinux - check our blog

Send Logs to Support

If there are any problems, collect the troubleshooting information in a tarball and send it to your customer support engineer.

  1. Tarball preparation to collect data for troubleshooting:
    a. Get all versions via:

    cat /opt/app_protect/VERSION > package_versions.txt<br>
    

    b. For CentOS:

    rpm -qa nginx-plus* app-protect* >> package_versions.txt<br>
    

    c. For Debian:

    cat /opt/app_protect/VERSION > package_versions.txt<br>
    

    d. Get OS via:

    cat /etc/os-release > system_version.txt && uname -r >> system_version.txt && cat /proc/version >> system_version.txt<br>
    
  2. Create a list of files for tarball in a file called logs.txt:

    • package_versions.txt
    • system_version.txt
    • /var/log/adm/*
    • /var/run/adm/*
    • /var/log/nginx/* (all NGINX files)
  3. Add all policies and log file configuration

  4. Add napd_shmem.log:

    admd -c > napd_shmem.log
    
  5. Add all NGINX configurations including all references such as /etc/nginx/nginx.conf

  6. Create the tarball:

    tar cvfz logs.tgz `cat logs.txt`
    
  7. Send logs.tgz to support.


This documentation applies to the following versions of NGINX App Protect DoS: 1.0.