NGINX App Protect WAF Troubleshooting Guide

Learn about the NGINX App Protect WAF Troubleshooting Guide.

Overview

This Troubleshooting Guide is intended to provide guidance to customers in the detection and correction of programming issues in NGINX App Protect. It may also be useful to IT in resolving any installation or configuration problems.

Refer to the below table for any NGINX App Protect WAF installation or configuration known problems.

Resolving Known Problems

Installation

Problem Solution
Starting version 3.12, installation steps and Docker deployment examples were changed in the Admin Guide. You may encounter one of the following error messages:

# example of yum installation error when the app-protect-security-updates repository is missing:
Error: Package: app-protect-compiler-1.234.0-1.el7.ngx.x86_64 (app-protect)
Requires: app-protect-attack-signatures
Error: Package: app-protect-compiler-1.234.0-1.el7.ngx.x86_64 (app-protect)
Requires: app-protect-threat-campaigns

# example of apt installation error when the app-protect-security-updates repository is missing:
The following packages have unmet dependencies:
app-protect-compiler : Depends: app-protect-attack-signatures
                       Depends: app-protect-threat-campaigns
Error: Unable to correct problems, you have held broken packages.
 Enable the app-protect-security-updates repository.

Configuration

Problem Solution
NGINX is not running (ps -aux)

Reloading NGINX fails		 Check the error log at /var/log/nginx/error.log
Fix the problem and re-run NGINX.
NGINX App Protect WAF functionality is not as expected NGINX App Protect WAF has several logs which can be used for troubleshooting.
Usually, it is best to look for any warning or error messages within the logs.
Refer to Logs Overview
Too many open files error message Increase number of file descriptors.
For example: worker_rlimit_nofile 65535; in the main context of nginx.conf file.
Refer to worker_rlimit_nofile directive
setrlimit ... failed (Permission denied) error message Increase the limit using the following command as the root user:
setsebool -P httpd_setrlimit 1;
Refer to Issue 4: Too many files are open Error
unknown directive app_protect_xxx error message App Protect module is not loaded. Add this line to the main (global) context of nginx.conf:
load_module "/etc/nginx/modules/ngx_http_app_protect_module.so";

ELK issues

ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for F5 App Protect WAF GitHub repo.

SELinux

App Protect files and processes are labeled with the following two contexts:

  • nap-compiler_t
  • nap-engine_t

NGINX Plus is labeled with the httpd_t context.

If you run into a situation where SELinux denies access to something, start the troubleshooting by searching for audit denials related to one of the above contexts.

For example:

ausearch --start recent -m avc --raw -se nap-engine_t

--start recent here means to start the search from 10 minutes ago

For more information about how to use NGINX Plus with SELinux - check our blog

Opening a Support Ticket

In order to open a support ticket, collect the troubleshooting information in a tarball and send it to your customer support engineer.

  1. Tarball preparation to collect data for troubleshooting:

    • Get all versions via:
    cat /opt/app_protect/VERSION /opt/app_protect/RELEASE > package_versions.txt

    For CentOS:

    rpm -qa nginx-plus* app-protect* >> package_versions.txt

    For Debian:

    apt list --installed | grep -E 'nginx-plus|app-protect' >> package_versions.txt
    • Get OS via:
    cat /etc/os-release > system_version.txt && uname -r >> system_version.txt && cat /proc/version >> system_version.txt

  2. Create a list of files for tarball in a file called logs.txt:

    • package_versions.txt
    • system_version.txt
    • /var/log/app_protect/* (all app protect files)
    • /var/log/nginx/* (all NGINX files)

  3. Add all policies and log file configuration

  4. Add all nginx configuration including all references such as /etc/nginx/nginx.conf

  5. Create the tarball:

    tar cvfz logs.tgz `cat logs.txt`

  6. Attach logs.tgz to support ticket.

  7. On the support ticket, in the NGINX App Protect WAF, set the release version according to the opt/app_protect/RELEASE file.