Troubleshooting Guide

Overview

Security logs (also known as Request logs or Traffic logs) contain information on HTTP requests and responses, how App Protect processes them, and the final decision made based on the configured policy parameters. The policy configuration defines the information contained in the Security log, such as whether requests are passed, blocked or alerted, due to violations, attack signatures, and other criteria.

App Protect uses its own logging mechanism for request logging rather than NGINX’s access logging mechanism (which is NGINX’s default logging mechanism).

The Security log has the following properties:

• Log Configuration: app_protect_security_log directive referencing security_log.json file

• Configuration contexts: nginx.conf: http, server, location

• File Destination? No. To use Syslog with files, you can redirect the log with Netcat and pipe:

  nc -vv -l “[ip]” “[port]” > “[name_of_file]” 2>&1

• Syslog Destination? Yes

Directives in nginx.conf

Security Log Configuration File

The file is in JSON format and consists of two parts:

1. filter: which requests are to be logged.
2. content: how the message is formatted.

{
    "filter": {
        "request_type": "illegal"
    },
    "content": {
        "format": "user-defined",
        "format_string": "client_ip=%ip_client%,client_port=%src_port%,request=%request%,violations=%violations%,signature_ids=%sig_ids%",
        "max_request_size": 2000,
        "max_message_size": "5k"
    }
}

{
    "filter": {
        "request_type": "all"
    },
    "content": {
        "format": "default",
        "max_request_size": "any",
        "max_message_size": "5k"
    }
}

{
    "filter": {
        "request_type": "illegal"
    },
    "content": {
        "format": "user-defined",
        "format_string": "Request ID %support_id%: %method% %uri% received on %date_time% from IP %ip_client% had the following violations: %violations%",
        "max_request_size": "any",
        "max_message_size": "5k"
    }
}

Syslog Transport

The syslog transport is over TCP. It is currently unsecured, meaning that SSL/TLS is not supported.

It is not guaranteed that all requests that match the filters will indeed reach their destination especially if the system is overwhelmed by the incoming traffic. In this case some log records may be dropped.

Factory Configuration Files

NGINX will provide example configuration files under /opt/app_protect/share/defaults/ with the following settings:

Available Security Log Attributes

The table below lists attributes that are generated in the security logs. When using customized logs (i.e., format=user-defined), you can add or remove entries from the list below.

Overview

The operation log consists of system operational and health events. The events are sent to the NGINX error log and are distinguished by the APP_PROTECT prefix followed by JSON body. The log level depends on the event: success is usually Notice while failure is Error. The timestamp is inherent in the error log.

Events

{
    "event": "waf_connected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true
}

{
    "event": "waf_connection_failure",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

|

{
    "event": "waf_disconnected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

{
    "event": "waf_resource_exception",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true,
    "resource": "cpu",
    "value": 98,
    "threshold": 95
}

{
    "event": "waf_resource_revert",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true,
    "resource": "cpu",
    "value": 88,
    "threshold": 90
}

{
    "event": "configuration_error",
    "error_message": "unknown argument",
    "line_number": 58
}

{
    "event": "configuration_load_start",
    "configSetFile": "/opt/app_protect/share/config_set.json"
}

{
    "event": configuration_load_failure",
    "error_message": "Failed to import Policy '/etc/nginx/default_policy.json' from '/etc/nginx/default_policy.json': Fail parse JSON Policy: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"xxxx\\nhdjk\\n\\n555\\n\") \n.\n",
    "error_line_number": 58
}

{
    "event": "configuration_load_success",
    "error_message": "csrfProtection/enabled cannot be set to true and set to false",
    "software_version": "1.3.42",
    "attack_signatures_package": {
        "version": "2020.02.20",
        "revision_datetime": "2020-02-20T13:35:38"
    }
}

log_format

This directive determines the format of the log messages using predefined variables. App Protect will enrich this set of variables with several security log attributes that are available to be included in the log_format. If log_format is not specified then the built-in format combined is used but, because that format does not include the extended App Protect variables, this directive must be used when the user wants to add App Protect information to the log.

access_log

This directive determines the destination of the access_log and the name of the format. The default is the file /etc/nginx/log/access.log using the combined format. In order to use the custom format that includes the NAP variables, use this directive with the name of the desired format.

App Protect Variables for Access Log

These are the variables added to Access Log. They are a subset of the Security log attributes. The Security log names are prefixed with $app_protect.

Note that many of the other Security log attributes that are not included here have exact or similar parallels among the NGINX variables also available for access log. For example, $request is parallel to the request security log attribute. See the full list of NGINX variables.

Example

http {
    log_format security_waf 'request_time=$request_time client_ip=$remote_addr,'
                             'request="$request", status=$status,'
                             'waf_policy=$app_protect_policy_name, waf_request_id=$app_protect_support_id'
                             'waf_action=$app_protect_outcome, waf_action_reason=$app_protect_outcome_reason';
 
    server {
 
        location / {
            access_log /etc/app_protect/logs/nginx-access.log security_waf;
            ...
        }
    }
}

Logger Configuration File

The logging configuration file is located in: /etc/app_protect/bd/logger.cfg and contains the App Protect modules for logging and debugging.

################################################################################################
#
#                                        Logger configuration file
#
#       Existing modules:
#
#       IO_PLUGIN (Requests & Responses) FTP_PLUGIN (ftp) SMTP_PLUGIN (smtp)
#       BEM (Accumulation Responses), ECARD (Tables),
#       ECARD_POLICY (Enforcer), BD_SSL (Communications), UMU (Memory),
#       IMF (Sockets), BD_MISC (Config and miscs),COOKIE_MGR (Cookies), REG_EXP (Regular expressions),
#       RESP_PARAMS (Extractions), ATTACK_SIG (Attack Signatures), BD_XML(XML Enforcer),
#       ATTACK_ENGINE (BF & BOT detect monitor), XML_PARSER (all xml engine), ACY (pattern match engine),
#       BD_PB (policy builder), BD_PB_SAMPLING (sampling decisions for pb), LEGAL_HASH (internal cache tables),
#       CLIENT_SIDE (Client Side infrastructre), STATS (policy builder statistics), ICAP (content inspection),
#       CLUSTER_ANOMALY (the anomaly distributed channel), PIPE (shmem channel bd-pbng, bd-lrn),
#       MPP_PARSER (Multipart parser), SA_PLUGIN (Session awareness), DATA_PROTECT (Data Protection Library),
#       GDM (Guardium DB security), ASM_IRULE (ASM iRule commands), LIBDATASYNC (Data Sync Library),
#       BD_CONF (BD MCP configuration), MPI_CHANNEL (BD initiated MPI events),
#       BD_FLUSH_TBLS(flush BD conf tables), CSRF (CSRF feature), BRUTE_FORCE_ENFORCER (Brute Force feature),
#       LONG_REQUEST (Long request), HTML_PARSER (HTML parser)
#
#       Log levels:
#
#       TS_DEBUG | TS_INFO | TS_NOTICE | TS_WARNING | TS_ERR | TS_CRIT
#
#       File numbers:
#
#       errors.log = 2 , debug.log = 3 (see /ts/agent/log.cfg)
#

Enabling Debug Logging

To add a module for logging:

#       MODULE = <module_name>;
#       LOG_LEVEL = <log level 1> | <log level 2> | ... | <log level n>;
#       FILE = <file number> (recommended 2 always);
#
#       Use # to comment out lines.

For example:

MODULE=IO_PLUGIN;
LOG_LEVEL=TS_INFO | TS_DEBUG;
FILE = 2;

MODULE = ALL;
LOG_LEVEL = TS_ERR | TS_CRIT | TS_WARNING | TS_NOTICE;
FILE=2;

Then run the perl script in the App Protect bin folder to begin log collections.

/bin/su -s /bin/bash -c '/opt/app_protect/bin/set_active.pl -g' nginx