NGINX Documentation

Troubleshooting Guide

App Protect Logging Overview

There are 3 types of logs that App Protect on NGINX generates:

  • Security log or Request log: The HTTP requests and how App Protect processed them, including violations and signatures found.
  • Operation log: Events such as startup, shutdown and reconfiguration.
  • Debug logs: technical messages at different levels of severity used to debug and figure out incidents and error behaviors.

Note that NGINX does not have audit logs in the sense of who did what. This can be done either from the orchestration system controlling NGINX (such as NGINX Controller) or by tracking the configuration files and the systemd invocations using Linux tools.

App Protect uses its own logging mechanism for request logging rather than NGINX’s access logging mechanism (which is NGINX’s default logging mechanism).

Type Log Configuration Configuration contexts File Destination Syslog Destination
Security app_protect_security_log directive referencing security_log.json file nginx.conf: http, server, location No Yes
Operation error_log directive, part of core NGINX nginx.conf - global Yes, NGINX error log Yes, NGINX error log
Debug /etc/app_protect/bd/logger.cfg. Log file name is the redirection in the invocation of the bd command line in the start script Global (not part of nginx.conf) Yes. Log file is in /var/log/app_protect default debug directory. No file rotation currently No

Security Logs

Overview

Security logs (also known as Request logs or Traffic logs) contain information on HTTP requests and responses, how App Protect processes them, and the final decision made based on the configured policy parameters. The policy configuration defines the information contained in the Security log, such as whether requests are passed, blocked or alerted, due to violations, attack signatures, and other criteria.

App Protect uses its own logging mechanism for request logging rather than NGINX’s access logging mechanism (which is NGINX’s default logging mechanism).

The Security log has the following properties:

  • Log Configuration: app_protect_security_log directive referencing security_log.json file

  • Configuration contexts: nginx.conf: http, server, location

  • File Destination? No. To use Syslog with files, you can redirect the log with Netcat and pipe:

    nc -vv -l “[ip]” “[port]” > “[name_of_file]” 2>&1

  • Syslog Destination? Yes

Directives in nginx.conf

app_protect_security_log_enable

This directive determines whether security logging will be enabled in the respective context.

The security log attributes are determined by the app_protect_security_log directive. The directive can be at the following contexts: http, server and location. When not present at a certain context, the directive is inherited from the context above it: location from server, then from http. If there is no directive at any of these context levels, then the logging is disabled for the respective context.

• Syntax: app_protect_security_log_enable on | off

• Levels: http, server, location

• Example: app_protect_security_log_enable on

Arguments
Argument Mandatory Meaning Default
ON-OFF Yes Whether to enable logging or not off

app_protect_security_log

The security log attributes are determined by the app_protect_security_log directive, if it was enabled in the respective context. The directive can be at the following context levels: http, server and location. When not present at a certain context, the directive is inherited from the context above it: location from server, then from http. If there is no directive at any of these context levels, but logging is enabled then the default is used for the respective context.

• Syntax: app_protect_security_log [LOG-CONFIG-FILE] [DESTINATION]

• Levels: http, server, location

• Example: app_protect_security_log “/opt/app_protect/share/defaults/log_illegal.json” syslog:server=mylogger.com:1234

Arguments
Argument Mandatory Meaning Default
LOG-CONFIG-FILE No The path to the log configuration file. See details below. Example of the default can be found in: /opt/app_protect/share/defaults/log_illegal.json
DESTINATION No The destination of the log messages in NGINX format. Only “syslog:” and IP/hostname options are accepted. Note: Hostname destination, except for localhost, is currently not supported in App Protect. syslog:server=localhost:514

Security Log Configuration File

The file is in JSON format and consists of two parts:

  1. filter: which requests are to be logged.
  2. content: how the message is formatted.

Filter

The filter is mandatory, although it may be left blank.

Element Meaning Type/Values Default
request_type Log according to what App Protect detected in the request.
Enumerated values:
  • all: all requests, both legal and illegal.
  • illegal: requests with violations (i.e., either alerted or blocked).
  • blocked: requests with violations that were blocked.
all

Content

This part of the configuration file specifies what will be logged, the format of the message, and size restrictions.

Content is mandatory. If the entire content field or any of its attributes are not defined, system-defined default values are used.

Examples

Log illegal requests in key-value format
{
    "filter": {
        "request_type": "illegal"
    },
    "content": {
        "format": "user-defined",
        "format_string": "client_ip=%ip_client%,client_port=%src_port%,request=%request%,violations=%violations%,signature_ids=%sig_ids%",
        "max_request_size": 2000,
        "max_message_size": "5k"
    }
}
Log state changing requests
{
    "filter": {
        "request_type": "all"
    },
    "content": {
        "format": "default",
        "max_request_size": "any",
        "max_message_size": "5k"
    }
}
A verbose custom formatted message
{
    "filter": {
        "request_type": "illegal"
    },
    "content": {
        "format": "user-defined",
        "format_string": "Request ID %support_id%: %method% %uri% received on %date_time% from IP %ip_client% had the following violations: %violations%",
        "max_request_size": "any",
        "max_message_size": "5k"
    }
}

NGINX Default Format String

When format = default, messages are shown in comma separated key-value pairs consisting of all the attributes appearing in Available Security Log Attributes.

The string starts like this:

“attack_type=“%attack_type%“,blocking_exception_reason=“%blocking_exception_reason%“,...”

Syslog Transport

The syslog transport is over TCP. It is currently unsecured, meaning that SSL/TLS is not supported.

It is not guaranteed that all requests that match the filters will indeed reach their destination especially if the system is overwhelmed by the incoming traffic. In this case some log records may be dropped.

Factory Configuration Files

NGINX will provide example configuration files under /opt/app_protect/share/defaults/ with the following settings:

Name Filter Content
log_illegal illegal requests format=default
log_f5_splunk illegal requests format=splunk, sizes are system-defined and cannot be changed.
log_f5_arcsight illegal requests format=arcsight, sizes are system-defined and cannot be changed.
log_all all format=default

Available Security Log Attributes

The table below lists attributes that are generated in the security logs. When using customized logs (i.e., format=user-defined), you can add or remove entries from the list below.

Attribute Name Description
attack_type A list of comma separated names of suspected attacks identified in a transaction.
blocking_exception_reason The blocking exception reason when a configured violation was not blocked.|
date_time The date and time the request was received by App Protect.
dest_port The port assigned to listen to incoming requests.
ip_client The source IP of the client initiating the request Note: if a proxy is being used, this may differ from the IP in the X-forwarded-for header.
is_truncated A flag that returns TRUE if a request is truncated in the security logs.
method The method of request. For example, GET, POST, HEAD.
policy_name The name of the App Protect policy for which the violation was triggered.
protocol The protocol used, either HTTP or HTTPS if terminating SSL on App Protect.
request The entire request including headers, query string, and data.
request_status
The status of client request made to Web Application as assigned by the App Protect policy. The possible values are:
  • blocked: The request was blocked due to a violation encountered. A blocking response page was returned to the client.
  • alerted: The request contains violation(s) but is not blocked (typical in cases where the enforcement mode is set to transparent).
  • passed: A successful request with no violations.
response_code The response code returned by the server.
severity The maximum severity calculated from all violations found in the request. It is a static value coming from the Violations.
sig_cves Signature CVEs value of the matched signatures.
sig_ids Signature ID value of the matching signature that resulted in the violation.
sig_names Signature name of the matching signature that resulted in the violation.
sig_set_names The signature set names of the matched signatures.
src_port The source port of the client.
sub_violations Refers to the sub-violations detected under the ‘HTTP protocol compliance failed’ and the ‘Evasion technique detected’ violations.
support_id A unique identifier for a transaction.
unit_hostname host name of the app-protect instance
uri The URI or Uniform Resource Identifier of the request.
violation_details XML including details about each violation.
violation_rating Estimation of the likelihood that the request is indeed a threat on a scale of 0 to 5: 0 - not a threat (no violations), 5 - most likely a threat
vs_name A unique identifier of the location in the nginx.conf file that this request is associated with. It contains the line number of the containing server block in nginx.conf, the server name, a numeric discriminator that distinguishes between multiple entries within the same server, and the location name. For example: ’34-mydomain.com:0-~/.*php(2).
x_forwarded_for_header_value X-Forwarded-For header information. This option is commonly used when proxies are involved to track the originator of the request.
outcome
One of the following:
  • PASSED: the request was sent to the backend server.
  • REJECTED: the request was blocked.
outcome_reason
One of the following:
  • SECURITY_WAF_OK: allowed with no violations (legal request).
  • SECURITY_WAF_VIOLATION: blocked due to security violations.
  • SECURITY_WAF_FLAGGED: allowed, although it has violations (illegal).
violations Comma-separated list of logical violation names (e.g. “VIOL_ATTACK_SIGNATURES,VIOL_HTTP_PROTOCOL”)

Operation Log

Overview

The operation log consists of system operational and health events. The events are sent to the NGINX error log and are distinguished by the APP_PROTECT prefix followed by JSON body. The log level depends on the event: success is usually Notice while failure is Error. The timestamp is inherent in the error log.

Events

Event Type Level Meaning
App Protect Connected Notice A worker successfully connected to NGINX App Protect Enforcer. The mode attribute should be operational unless there is an ongoing problem.
{
    "event": "waf_connected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true
}
Event Type Level Meaning
App Protect Connection Failure Error A worker attempted to connect to NGINX App Protect but the operation failed. The mode should be failure.
{
    "event": "waf_connection_failure",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}

|

Event Type Level Meaning
App Protect Disconnected Error Engine disconnected from Worker (socket closed). The mode should be failure.
{
    "event": "waf_disconnected",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true
}
Event Type Level Meaning
App Protect Resource Exception Warning Resource, as measured by the Worker, exceeded limits (above high threshold). Mode should be failure. It may have already been in this mode because there are other resources that had exceeded their limits.
{
    "event": "waf_resource_exception",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "failure",
    "mode_changed": true,
    "resource": "cpu",
    "value": 98,
    "threshold": 95
}
Event Type Level Meaning
App Protect Resource Reverted to Normal Warning Resource, as measured by the Worker, went back to normal range (below low threshold). Mode should be operational, unless there are other resources which are still out of limits.
{
    "event": "waf_resource_revert",
    "bd_thread_id": 3,
    "worker_pid": 4928,
    "mode": "operational",
    "mode_changed": true,
    "resource": "cpu",
    "value": 88,
    "threshold": 90
}
Event Type Level Meaning
Configuration Error Error There were errors in the AppProtect directives in the nginx.conf file. This is issued if the directive was spelled correctly, otherwise NGINX core will issue an error. This event occurs before configuration_load_start and means there will be no configuration load. This event is generated only on configuration reload. It cannot be generated on first configuration as there is no error log configured yet.
{
    "event": "configuration_error",
    "error_message": "unknown argument",
    "line_number": 58
}
Event Type Level Meaning
Configuration Load Start Notice App Protect configuration load process started. The configuration consists of all the policies, security log configurations and global settings. These all are part of the config set file generated by the module and passed to the Policy Compiler. The path to this file in included in the event message. This event is generated only on configuration reload. It cannot be generated on first configuration as there is no error log configured yet.
{
    "event": "configuration_load_start",
    "configSetFile": "/opt/app_protect/share/config_set.json"
}
Event Type Level Meaning
Configuration Load Failure Error There was an error in one of the configuration files: file not found, failed to compile, or the configuration failed to load to the engine.
{
    "event": configuration_load_failure",
    "error_message": "Failed to import Policy '/etc/nginx/default_policy.json' from '/etc/nginx/default_policy.json': Fail parse JSON Policy: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before \"xxxx\\nhdjk\\n\\n555\\n\") \n.\n",
    "error_line_number": 58
}
Event Type Level Meaning
Configuration Load Success Notice The WAF configuration process ended successfully: all policies, log configuration and global settings were loaded to NGINX App Protect and all traffic will be handled by this configuration. The “error_message” contains warnings. This event is generated also on the initial configuration (when NGINX starts). Also includes the signature update version which reflects the date the package was released and the exact revision time in datetime format that also includes the time of day, thus compatible with the revision date time in the WAF policy signature-requirements element.
{
    "event": "configuration_load_success",
    "error_message": "csrfProtection/enabled cannot be set to true and set to false",
    "software_version": "1.3.42",
    "attack_signatures_package": {
        "version": "2020.02.20",
        "revision_datetime": "2020-02-20T13:35:38"
    }
}

Access Logs

Access log is NGINX’s request log mechanism. It is controlled by two directives.

log_format

This directive determines the format of the log messages using predefined variables. App Protect will enrich this set of variables with several security log attributes that are available to be included in the log_format. If log_format is not specified then the built-in format combined is used but, because that format does not include the extended App Protect variables, this directive must be used when the user wants to add App Protect information to the log.

access_log

This directive determines the destination of the access_log and the name of the format. The default is the file /etc/nginx/log/access.log using the combined format. In order to use the custom format that includes the NAP variables, use this directive with the name of the desired format.

App Protect Variables for Access Log

These are the variables added to Access Log. They are a subset of the Security log attributes. The Security log names are prefixed with $app_protect.

Name Meaning Comment
$app_protect_support_id Unique ID assigned to the request by App Protect. To be used to correlate the access log with the security log. Left empty in failure mode.
$app_protect_outcome
One of:
  • PASSED: request was sent to origin server.
  • REJECTED: request was blocked.
 
$app_protect_outcome_reason
One of:
  • SECURITY_WAF_OK: allowed with no violations (legal request).
  • SECURITY_WAF_VIOLATION: blocked due to security violations.
  • SECURITY_WAF_FLAGGED: allowed although it has violations (illegal).
  • SECURITY_WAF_BYPASS: WAF was supposed to inspect the request but it didn’t (because of unavailability or resource shortage). The request was PASSED or REJECTED according to the failure mode action determined by the user.
 
$app_protect_policy_name The name of the policy that enforced the request.  
$app_protect_version The App Protect version string: major.minor.build format. Does not include the NGINX plus version (e.g. R21). The latter is available in $version variable.

Note that many of the other Security log attributes that are not included here have exact or similar parallels among the NGINX variables also available for access log. For example, $request is parallel to the request security log attribute. See the full list of NGINX variables.

Example

http {
    log_format security_waf 'request_time=$request_time client_ip=$remote_addr,'
                             'request="$request", status=$status,'
                             'waf_policy=$app_protect_policy_name, waf_request_id=$app_protect_support_id'
                             'waf_action=$app_protect_outcome, waf_action_reason=$app_protect_outcome_reason';
 
    server {
 
        location / {
            access_log /etc/app_protect/logs/nginx-access.log security_waf;
            ...
        }
    }
}

Debug Logs

Debug log settings determine the minimum log level and the internal App Protect components included in the log. We include a perl script that allows modification of those parameters without having to restart App Protect or reload NGINX.

nginx.conf does not refer to the NGINX App Protect debug log configuration neither directly nor indirectly.

Logger Configuration File

The logging configuration file is located in: /etc/app_protect/bd/logger.cfg and contains the App Protect modules for logging and debugging.

################################################################################################
#
#                                        Logger configuration file
#
#       Existing modules:
#
#       IO_PLUGIN (Requests & Responses) FTP_PLUGIN (ftp) SMTP_PLUGIN (smtp)
#       BEM (Accumulation Responses), ECARD (Tables),
#       ECARD_POLICY (Enforcer), BD_SSL (Communications), UMU (Memory),
#       IMF (Sockets), BD_MISC (Config and miscs),COOKIE_MGR (Cookies), REG_EXP (Regular expressions),
#       RESP_PARAMS (Extractions), ATTACK_SIG (Attack Signatures), BD_XML(XML Enforcer),
#       ATTACK_ENGINE (BF & BOT detect monitor), XML_PARSER (all xml engine), ACY (pattern match engine),
#       BD_PB (policy builder), BD_PB_SAMPLING (sampling decisions for pb), LEGAL_HASH (internal cache tables),
#       CLIENT_SIDE (Client Side infrastructre), STATS (policy builder statistics), ICAP (content inspection),
#       CLUSTER_ANOMALY (the anomaly distributed channel), PIPE (shmem channel bd-pbng, bd-lrn),
#       MPP_PARSER (Multipart parser), SA_PLUGIN (Session awareness), DATA_PROTECT (Data Protection Library),
#       GDM (Guardium DB security), ASM_IRULE (ASM iRule commands), LIBDATASYNC (Data Sync Library),
#       BD_CONF (BD MCP configuration), MPI_CHANNEL (BD initiated MPI events),
#       BD_FLUSH_TBLS(flush BD conf tables), CSRF (CSRF feature), BRUTE_FORCE_ENFORCER (Brute Force feature),
#       LONG_REQUEST (Long request), HTML_PARSER (HTML parser)
#
#       Log levels:
#
#       TS_DEBUG | TS_INFO | TS_NOTICE | TS_WARNING | TS_ERR | TS_CRIT
#
#       File numbers:
#
#       errors.log = 2 , debug.log = 3 (see /ts/agent/log.cfg)
#

Enabling Debug Logging

To add a module for logging:

#       MODULE = <module_name>;
#       LOG_LEVEL = <log level 1> | <log level 2> | ... | <log level n>;
#       FILE = <file number> (recommended 2 always);
#
#       Use # to comment out lines.

For example:

MODULE=IO_PLUGIN;
LOG_LEVEL=TS_INFO | TS_DEBUG;
FILE = 2;
MODULE = ALL;
LOG_LEVEL = TS_ERR | TS_CRIT | TS_WARNING | TS_NOTICE;
FILE=2;

Then run the perl script in the App Protect bin folder to begin log collections.

/bin/su -s /bin/bash -c '/opt/app_protect/bin/set_active.pl -g' nginx

ELK issues

ELK issues are addressed directly in GitHub by posting the issue to Kibana dashboards for F5 App Protect WAF GitHub repo.