Troubleshooting Guide

App Protect Troubleshooting Overview

If there are any problems collect the troubleshooting information in a tarball and send it to your customer support engineer.

1. Tarball preparation to collect data for troubleshooting:

   • Get all versions via:

   cat /opt/app_protect/VERSION > package_versions.txt
   

   For CentOS:

   rpm -qa nginx-plus* app-protect* >> package_versions.txt
   

   For Debian:

   apt list --installed | grep -E 'nginx-plus|app-protect' >> package_versions.txt
   

   • Get OS via:

   cat /etc/os-release > system_version.txt && uname -r >> system_version.txt && cat /proc/version >> system_version.txt
   

2. Create a list of files for tarball in a file called logs.txt:

   • package_versions.txt
   • system_version.txt
   • /var/log/app_protect/* (all app protect files)
   • /var/log/nginx/* (all NGINX files)

3. Add all policies and log file configuration

4. Add all nginx configuration including all references such as /etc/nginx/nginx.conf

5. Create the tarball:

   tar cvfz logs.tgz `cat logs.txt`
   

6. Send logs.tgz to support.

App Protect Logging Overview

There are 3 types of logs that App Protect on NGINX generates:

• Security log or Request log: The HTTP requests and how App Protect processed them, including violations and signatures found.
• Operation log: Events such as startup, shutdown and reconfiguration.
• Debug logs: technical messages at different levels of severity used to debug and resolve incidents and error behaviors.

Note that NGINX does not have audit logs in the sense of who did what. This can be done either from the orchestration system controlling NGINX (such as NGINX Controller) or by tracking the configuration files and the systemd invocations using Linux tools.

App Protect uses its own logging mechanism for request logging rather than NGINX’s access logging mechanism (which is NGINX’s default logging mechanism).

Operation Logs

{
    "event": "waf_connected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true
}

{
    "event": "waf_connection_failure",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

{
    "event": "waf_disconnected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

{
    "event": "waf_resource_exception",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true,
    "resource": "cpu",
    "value": 98,
    "threshold": 95
}

{
    "event": "waf_resource_revert",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true,
    "resource": "cpu",
    "value": 88,
    "threshold": 90
}

{
    "event": "configuration_error",
    "error_message": "unknown argument",
    "line_number": 58
}

{
    "event": "configuration_load_start",
    "configSetFile": "/opt/app_protect/share/config_set.json"
}

{
    "event": configuration_load_failure",
    "error_message": "Failed to import Policy '/etc/nginx/default_policy.json' from '/etc/nginx/default_policy.json': Fail parse JSON Policy: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"xxxx\\nhdjk\\n\\n555\\n\") \n.\n",
    "error_line_number": 58
}

{
    "event": "configuration_load_success",
    "error_message": "csrfProtection/enabled cannot be set to true and set to false",
    "software_version": "1.3.42",
    "attack_signatures_package": {
        "version": "2020.02.20",
        "revision_datetime": "2020-02-20T13:35:38"
    }
}

Access Logs

Access log is NGINX’s request log mechanism. It is controlled by two directives.

http {
    log_format security_waf 'request_time=$request_time client_ip=$remote_addr,'
                             'request="$request", status=$status,'
                             'waf_policy=$app_protect_policy_name, waf_request_id=$app_protect_support_id'
                             'waf_action=$app_protect_outcome, waf_action_reason=$app_protect_outcome_reason';

    server {

        location / {
            access_log /etc/app_protect/logs/nginx-access.log security_waf;
            ...
        }
    }
}

Debug Logs

Debug log settings determine the minimum log level and the internal App Protect components included in the log. We include a perl script that allows modification of those parameters without having to restart App Protect or reload NGINX.

nginx.conf does not refer to the NGINX App Protect debug log configuration neither directly nor indirectly.

################################################################################################
#
#                                        Logger configuration file
#
#       Existing modules:
#
#       IO_PLUGIN (Requests & Responses) FTP_PLUGIN (ftp) SMTP_PLUGIN (smtp)
#       BEM (Accumulation Responses), ECARD (Tables),
#       ECARD_POLICY (Enforcer), BD_SSL (Communications), UMU (Memory),
#       IMF (Sockets), BD_MISC (Config and miscs),COOKIE_MGR (Cookies), REG_EXP (Regular expressions),
#       RESP_PARAMS (Extractions), ATTACK_SIG (Attack Signatures), BD_XML(XML Enforcer),
#       ATTACK_ENGINE (BF & BOT detect monitor), XML_PARSER (all xml engine), ACY (pattern match engine),
#       BD_PB (policy builder), BD_PB_SAMPLING (sampling decisions for pb), LEGAL_HASH (internal cache tables),
#       CLIENT_SIDE (Client Side infrastructure), STATS (policy builder statistics), ICAP (content inspection),
#       CLUSTER_ANOMALY (the anomaly distributed channel), PIPE (shmem channel bd-pbng, bd-lrn),
#       MPP_PARSER (Multipart parser), SA_PLUGIN (Session awareness), DATA_PROTECT (Data Protection Library),
#       GDM (Guardium DB security), ASM_IRULE (ASM iRule commands), LIBDATASYNC (Data Sync Library),
#       BD_CONF (BD MCP configuration), MPI_CHANNEL (BD initiated MPI events),
#       BD_FLUSH_TBLS(flush BD conf tables), CSRF (CSRF feature), BRUTE_FORCE_ENFORCER (Brute Force feature),
#       LONG_REQUEST (Long request), HTML_PARSER (HTML parser)
#
#       Log levels:
#
#       TS_DEBUG | TS_INFO | TS_NOTICE | TS_WARNING | TS_ERR | TS_CRIT
#
#       File numbers:
#
#       errors.log = 2 , debug.log = 3 (see /ts/agent/log.cfg)
#

#       MODULE = <module_name>;
#       LOG_LEVEL = <log level 1> | <log level 2> | ... | <log level n>;
#       FILE = <file number> (recommended 2 always);
#
#       Use # to comment out lines.

MODULE=IO_PLUGIN;
LOG_LEVEL=TS_INFO | TS_DEBUG;
FILE = 2;

MODULE = ALL;
LOG_LEVEL = TS_ERR | TS_CRIT | TS_WARNING | TS_NOTICE;
FILE=2;

/bin/su -s /bin/bash -c '/opt/app_protect/bin/set_active.pl -g' nginx

ELK issues

ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for F5 App Protect WAF GitHub repo.

SELinux

App Protect files and processes are labeled with the following two contexts:

• nap-compiler_t
• nap-engine_t

NGINX Plus is labeled with the httpd_t context.

If you run into a situation where SELinux denies access to something, start the troubleshooting by searching for audit denials related to one of the above contexts.

For example:

ausearch --start recent -m avc --raw -se nap-engine_t

--start recent here means to start the search from 10 minutes ago

For more information about how to use NGINX Plus with SELinux - check our blog