Troubleshooting Guide

App Protect Troubleshooting Overview

If there are any problems collect the troubleshooting information in a tarball and send it to your customer support engineer.

1. Tarball preparation to collect data for troubleshooting:

   • Get all versions via:

   cat /opt/app_protect/VERSION > package_versions.txt
   

   For CentOS:

   rpm -qa nginx-plus* app-protect* >> package_versions.txt
   

   For Debian:

   apt list --installed | grep -E 'nginx-plus|app-protect' >> package_versions.txt
   

   • Get OS via:

   cat /etc/os-release > system_version.txt && uname -r >> system_version.txt && cat /proc/version >> system_version.txt
   

2. Create a list of files for tarball in a file called logs.txt:

   • package_versions.txt
   • system_version.txt
   • /var/log/app_protect/* (all app protect files)
   • /var/log/nginx/* (all NGINX files)

3. Add all policies and log file configuration

4. Add all nginx configuration including all references such as /etc/nginx/nginx.conf

5. Create the tarball:

   tar cvfz logs.tgz `cat logs.txt`
   

6. Send logs.tgz to support.

App Protect Logging Overview

There are 3 types of logs that App Protect on NGINX generates:

• Security log or Request log : The HTTP requests and how App Protect processed them, including violations and signatures found.
• Operation log : Events such as startup, shutdown and reconfiguration.
• Debug logs : technical messages at different levels of severity used to debug and resolve incidents and error behaviors.

Note that NGINX does not have audit logs in the sense of who did what. This can be done either from the orchestration system controlling NGINX (such as NGINX Controller) or by tracking the configuration files and the systemd invocations using Linux tools.

App Protect uses its own logging mechanism for request logging rather than NGINX’s access logging mechanism (which is NGINX’s default logging mechanism).

Operation Logs

Overview

The operation logs consists of system operational and health events. The events are sent to the NGINX error log and are distinguished by the APP_PROTECT prefix followed by JSON body. The log level depends on the event: success is usually Notice while failure is Error. The timestamp is inherent in the error log.

Events

{
    "event": "waf_connected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true
}

{
    "event": "waf_connection_failure",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

|

{
    "event": "waf_disconnected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

{
    "event": "waf_resource_exception",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true,
    "resource": "cpu",
    "value": 98,
    "threshold": 95
}

{
    "event": "waf_resource_revert",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true,
    "resource": "cpu",
    "value": 88,
    "threshold": 90
}

{
    "event": "configuration_error",
    "error_message": "unknown argument",
    "line_number": 58
}

{
    "event": "configuration_load_start",
    "configSetFile": "/opt/app_protect/share/config_set.json"
}

{
    "event": configuration_load_failure",
    "error_message": "Failed to import Policy '/etc/nginx/default_policy.json' from '/etc/nginx/default_policy.json': Fail parse JSON Policy: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"xxxx\\nhdjk\\n\\n555\\n\") \n.\n",
    "error_line_number": 58
}

{
    "event": "configuration_load_success",
    "error_message": "csrfProtection/enabled cannot be set to true and set to false",
    "software_version": "1.3.42",
    "attack_signatures_package": {
        "version": "2020.02.20",
        "revision_datetime": "2020-02-20T13:35:38"
    }
}

Access Logs

Access log is NGINX’s request log mechanism. It is controlled by two directives.

log_format

This directive determines the format of the log messages using predefined variables. App Protect will enrich this set of variables with several security log attributes that are available to be included in the log_format. If log_format is not specified then the built-in format combined is used but, because that format does not include the extended App Protect variables, this directive must be used when the user wants to add App Protect information to the log.

access_log

This directive determines the destination of the access_log and the name of the format. The default is the file /etc/nginx/log/access.log using the combined format. In order to use the custom format that includes the NAP variables, use this directive with the name of the desired format.

App Protect Variables for Access Log

These are the variables added to Access Log. They are a subset of the Security log attributes. The Security log names are prefixed with $app_protect.

Note that many of the other Security log attributes that are not included here have exact or similar parallels among the NGINX variables also available for access log. For example, $request is parallel to the request security log attribute. See the full list of NGINX variables.

Example

http {
    log_format security_waf 'request_time=$request_time client_ip=$remote_addr,'
                             'request="$request", status=$status,'
                             'waf_policy=$app_protect_policy_name, waf_request_id=$app_protect_support_id'
                             'waf_action=$app_protect_outcome, waf_action_reason=$app_protect_outcome_reason';

    server {

        location / {
            access_log /etc/app_protect/logs/nginx-access.log security_waf;
            ...
        }
    }
}

Debug Logs

Debug log settings determine the minimum log level and the internal App Protect components included in the log. We include a perl script that allows modification of those parameters without having to restart App Protect or reload NGINX.

nginx.conf does not refer to the NGINX App Protect debug log configuration neither directly nor indirectly.

Logger Configuration File

The logging configuration file is located in: /etc/app_protect/bd/logger.cfg and contains the App Protect modules for logging and debugging.

################################################################################################
#
#                                        Logger configuration file
#
#       Existing modules:
#
#       IO_PLUGIN (Requests & Responses) FTP_PLUGIN (ftp) SMTP_PLUGIN (smtp)
#       BEM (Accumulation Responses), ECARD (Tables),
#       ECARD_POLICY (Enforcer), BD_SSL (Communications), UMU (Memory),
#       IMF (Sockets), BD_MISC (Config and miscs),COOKIE_MGR (Cookies), REG_EXP (Regular expressions),
#       RESP_PARAMS (Extractions), ATTACK_SIG (Attack Signatures), BD_XML(XML Enforcer),
#       ATTACK_ENGINE (BF & BOT detect monitor), XML_PARSER (all xml engine), ACY (pattern match engine),
#       BD_PB (policy builder), BD_PB_SAMPLING (sampling decisions for pb), LEGAL_HASH (internal cache tables),
#       CLIENT_SIDE (Client Side infrastructure), STATS (policy builder statistics), ICAP (content inspection),
#       CLUSTER_ANOMALY (the anomaly distributed channel), PIPE (shmem channel bd-pbng, bd-lrn),
#       MPP_PARSER (Multipart parser), SA_PLUGIN (Session awareness), DATA_PROTECT (Data Protection Library),
#       GDM (Guardium DB security), ASM_IRULE (ASM iRule commands), LIBDATASYNC (Data Sync Library),
#       BD_CONF (BD MCP configuration), MPI_CHANNEL (BD initiated MPI events),
#       BD_FLUSH_TBLS(flush BD conf tables), CSRF (CSRF feature), BRUTE_FORCE_ENFORCER (Brute Force feature),
#       LONG_REQUEST (Long request), HTML_PARSER (HTML parser)
#
#       Log levels:
#
#       TS_DEBUG | TS_INFO | TS_NOTICE | TS_WARNING | TS_ERR | TS_CRIT
#
#       File numbers:
#
#       errors.log = 2 , debug.log = 3 (see /ts/agent/log.cfg)
#

Enabling Debug Logging

To add a module for logging:

#       MODULE = <module_name>;
#       LOG_LEVEL = <log level 1> | <log level 2> | ... | <log level n>;
#       FILE = <file number> (recommended 2 always);
#
#       Use # to comment out lines.

For example:

MODULE=IO_PLUGIN;
LOG_LEVEL=TS_INFO | TS_DEBUG;
FILE = 2;

MODULE = ALL;
LOG_LEVEL = TS_ERR | TS_CRIT | TS_WARNING | TS_NOTICE;
FILE=2;

Then run the perl script in the App Protect bin folder to begin log collections.

/bin/su -s /bin/bash -c '/opt/app_protect/bin/set_active.pl -g' nginx

ELK issues

ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for F5 App Protect WAF GitHub repo .

SELinux

App Protect files and processes are labeled with the following two contexts:

• nap-compiler_t
• nap-engine_t

NGINX Plus is labeled with the httpd_t context.

If you run into a situation where SELinux denies access to something, start the troubleshooting by searching for audit denials related to one of the above contexts.

For example:

ausearch --start recent -m avc --raw -se nap-engine_t

--start recent here means to start the search from 10 minutes ago

For more information about how to use NGINX Plus with SELinux - check our blog