Splunk Enterprise via Fluentd¶

Example overview¶

Webhooks can be used as system log sources. The number of log sources depends on the system complexity: the more components in the system, the greater number of log sources and logs. The most common logging scheme in complex systems consists of the following components:

Log collector: accepts logs from several sources and forwards logs to the SIEM system
SIEM system: used to analyze logs and monitor the system status

In the provided example, events are sent via webhooks to the Fluentd log collector and forwarded to the Splunk SIEM system.

Used resources¶

Splunk Enterprise with WEB URL https://109.111.35.11:8000 and API URL https://109.111.35.11:8088
Fluentd installed on Debian 10.4 (Buster) and available by https://fluentd‑example‑domain.com
Administrator access to Wallarm Console in EU cloud to configure the webhook integration

Splunk Enterprise configuration¶

Fluentd logs are sent to Splunk HTTP Event Controller with the name Wallarm Fluentd logs and other default settings:

To access the HTTP Event Controller, generated token f44b3179-91aa-44f5-a6f7-202265e10475 will be used.

More detailed description of Splunk HTTP Event Controller setup is available in the official Splunk documentation.

Fluentd configuration¶

Fluentd is configured in the td-agent.conf file:

Incoming webhook processing is configured in the source directive:
- All HTTP and HTTPS traffic is sent to 9880 Fluentd port
- TLS certificate for HTTPS connection is located in the file /etc/pki/ca.pem
Forwarding logs to Splunk and log output are configured in the match directive:
- All event logs are copied from Fluentd and forwarded to Splunk HTTP Event Controller via the output plugin fluent-plugin-splunk-hec
- Fluentd logs are additionally printed on the command line in JSON format (19-22 code lines). The setting is used to verify that events are logged via Fluentd

<source>
  @type http # input plugin for HTTP and HTTPS traffic
  port 9880 # port for incoming requests
  <transport tls> # certificates for HTTPS connection
    ca_path /etc/pki/ca.pem
  </transport>
</source>
<match **>
  @type copy
  <store>
      @type splunk_hec # output plugin fluent-plugin-splunk-hec to forward logs to Splunk API via HTTP Event Controller
      hec_host 109.111.35.11 # Splunk host
      hec_port 8088 # Splunk API port
      hec_token f44b3179-91aa-44f5-a6f7-202265e10475 # HTTP Event Controller token
    <format>
      @type json # format of forwarded logs
    </format>
  </store>
  <store>
     @type stdout # output plugin to print Fluentd logs on the command line
     output_type json # format of logs printed on the command line
  </store>
</match>

More detailed description of configuration files is available in the official Fluentd documentation.

Testing Fluentd configuration

To check that Fluentd logs are created and forwarded to Splunk, the PUT or POST request can be sent to Fluentd.

Request example:

curl -X POST 'https://fluentd‑example‑domain.com' -H "Content-Type: application/json" -H "Authorization: Splunk f44b3179-91aa-44f5-a6f7-202265e10475" -d '{"key1":"value1", "key2":"value2"}'

Fluentd logs:

Splunk logs:

Configuration of webhook integration¶

Webhooks are sent to https://fluentd-example-domain.com
Webhooks are sent via POST requests POST
Additional authentication parameter X-Auth-Token is passed in the request
Webhooks send to Webhook URL all available events: hits, system events, vulnerabilities, scope changes

Example testing¶

To test the configuration, a new user is added in Wallarm Console:

Fluentd will log the event as follows:

Log about new user in Splunk from Fluentd

The following entry will be displayed in Splunk events: