Skip to main content

Monitoring

Logging​

By default, Gate emits logs to the stderr sink in json format with log level info. If not specified, the default configuration is equivalent to the following:

GATE_LOG_FORMAT=json
GATE_LOG_LEVEL=info
GATE_LOG_OUTPUTS_0_SINK=stderr

Supported log levels:

  • panic
  • fatal
  • error
  • warn
  • info
  • debug
  • trace
caution

Trace and debug logs may contain sensitive information from requests (like headers). Please make sure to set the log-level to debug only in environments where you can safely print the configuration.

Supported log formats:

  • text
  • json

Supported sinks:

  • stderr
  • stdout
  • syslog

Multiple outputs can be configured simultaneously, and each one can specify a different sink, log format and log level overriding the defaults. Refer to the following sections for logging to standard streams or SysLog

Logging to standard streams​

Gate supports logging to the stdout and stderr standard streams. For example, the following configuration outputs trace and higher severity logs (i.e., all logs) to stdout in textual format, and only warnings and higher severity to stderr as JSON:

GATE_LOG_OUTPUTS_0_SINK=stdout
GATE_LOG_OUTPUTS_0_FORMAT=text
GATE_LOG_OUTPUTS_0_LEVEL=trace

GATE_LOG_OUTPUTS_1_SINK=stderr
GATE_LOG_OUTPUTS_1_FORMAT=json
GATE_LOG_OUTPUTS_1_LEVEL=warn

The format and level configuration parameters are optional for outputs and default to the top-level log.format and log.level values.

Buffering​

By default, the output of the Gate process's standard streams is "unbuffered", only relying on the OS-provided stream buffers. If your application produces logs to standard streams faster than your infrastructure or collectors can consume, you can configure Gate to provide its own buffering by using the sink-specific parameters.buffer_size:

GATE_LOG_OUTPUTS_0_SINK=stdout
GATE_LOG_OUTPUTS_0_FORMAT=text
GATE_LOG_OUTPUTS_0_LEVEL=trace
GATE_LOG_OUTPUTS_0_PARAMETERS_BUFFER_SIZE=65536

In the example above a stdout sink has an extra Gate-managed buffer of 64KiB.

Logging to SysLog​

Gate supports shipping logs to a RFC5424-compliant SysLog server by using the syslog sink type. These are the supported configuration values:

GATE_LOG_OUTPUTS_0_SINK=syslog
GATE_LOG_OUTPUTS_0_FORMAT=<Log format>
GATE_LOG_OUTPUTS_0_LEVEL=<Log level>
GATE_LOG_OUTPUTS_0_PARAMETERS_NETWORK=<Network>
GATE_LOG_OUTPUTS_0_PARAMETERS_ADDRESS=<Address>
GATE_LOG_OUTPUTS_0_PARAMETERS_FACILITY=<SysLog facility>
GATE_LOG_OUTPUTS_0_PARAMETERS_TAG=<SysLog tag>

where:

  • <Log format> supports the same values as all other sinks: trace, debug, info, warn, error, fatal, panic
  • <Log level> supports the same values as all other sinks: text, json
  • <Network> is the network type to connect to the SysLog server: tcp, tcp4 (IPv4-only), tcp6 (IPv6-only), udp, udp4 (IPv4-only), udp6
  • <Address> is the network address of the remote SysLog server in the host:port format (e.g., syslog.example.com:6514)
  • <SysLog facility> is the SysLog facility value: between 0 and 23 (included)
  • <SysLog tag> is the SysLog application tag (also referred to as application name): any string value identifying the current Gate instance

By default, Gate connects to the specified remote SysLog server over TLS (v1.2 and above). You can optionally instruct Gate to skip certificate verification or disable TLS entirely by using the tls.insecure and tls.disabled boolean parameters, respectively:

GATE_LOG_OUTPUTS_0_SINK=syslog
...
GATE_LOG_OUTPUTS_0_PARAMETERS_TLS_INSECURE=<Skip certificate verification>
GATE_LOG_OUTPUTS_0_PARAMETERS_TLS_DISABLED=<Disable TLS>

Priority​

The resulting SysLog priority value of each message is a combination for its "facility" value, as provided in the configuration, and the "severity" value of each log line. In particular, Gate maps log levels to SysLog severity values as follows:

  • panic, fatal: Critical
  • error: Error
  • warn: Warning
  • info: Informational
  • debug, trace: Debug

Connection management​

Gate automatically manages the connection to the remote SysLog server. In particular, it will try to connect immediately on start-up and fail to initialize if a connection cannot be established. If a connection or communication error is detected, Gate will automatically try to reconnect, with an exponential backoff delay. Gate continues to log without interruptions even while not connected to the SysLog server, as described in the message queueing section.

Message Queueing​

To avoid a negative impact on performance due to slow or intermittent connectivity with the SysLog server, Gate's syslog sink handles logs asynchronously. Any logging operation simply adds the log entry to a transmission queue. The log is then asynchronously sent to the SysLog server as soon as a connection becomes available. Please refer to the connection management section for details.

The transmission queue holds up to 64K log entries, but it may fill up entirely under certain circumstances, such as a prolonged disconnection from the SysLog server, or the server being consistently slow at ingesting messages. In this case, Gate prioritizes continuity of operations, dropping the oldest message in the queue in favor of each new entry.

Metadata​

Gate populates the following fields for all outgoing SysLog messages:

  • HOSTNAME: hostname of the machine as provided by the OS
  • APP-NAME: application tag as specified in the syslog sink configuration parameters
  • PROCID: the PID of the Gate process
  • TIMESTAMP: current time with micro-second precision, the maximum allowed by the SysLog protocol

On top of the information provided in the header, Gate takes advantage of the SysLog Structured Data mechanism to enrich each message with the following extra information:

Health checks​

ExtAuth mode​

Gate implements the gRPC Health Checking Protocol v1. We also include grpc_health_probe in all our docker images.

Here's an example of a Docker Compose setup:

version: "3"
services:
  gate:
    image: slashid/gate-free:latest
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "/usr/local/bin/grpc_health_probe", "-addr=:8080"]
      interval: 5s
      timeout: 10s
      retries: 3

Here's an example of a Kubernetes v1.23+ setup using Kubernetes' built-in gRPC health checking capability:

apiVersion: v1
kind: Pod
metadata:
  name: gate
spec:
  containers:
    - name: gate
      image: slashid/gate-free:latest
      ports:
        - containerPort: 8080
      livenessProbe:
        grpc:
          port: 8080
        initialDelaySeconds: 5

Finally, an example of a Kubernetes setup using grpc_health_probe:

apiVersion: v1
kind: Pod
metadata:
  name: gate
spec:
  containers:
    - name: gate
      image: slashid/gate-free:latest
      ports:
        - containerPort: 8080
      readinessProbe:
        exec:
          command: ["/usr/local/bin/grpc_health_probe", "-addr=:8080"]
        initialDelaySeconds: 5
      livenessProbe:
        exec:
          command: ["/usr/local/bin/grpc_health_probe", "-addr=:8080"]
        initialDelaySeconds: 10