Skip to content

Micro Focus ArcSight Logger via Fluentd

Example overview

Webhooks can be used as system log sources. The number of log sources depends on the system complexity: the more components in the system, the greater number of log sources and logs. The most common logging scheme in complex systems consists of the following components:

  • Log collector: accepts logs from several sources and forwards logs to the SIEM system

  • SIEM system: used to analyze logs and monitor the system status

In the provided example, events are sent via webhooks to the Fluentd log collector and forwarded to the ArcSight Logger system.

Webhook flow

Integration with the Enterprise version of ArcSight ESM

To configure forwarding logs from Fluentd to the Enterprise version of ArcSight ESM, it is recommended to configure the Syslog Connector on the ArcSight side and then forward logs from Fluentd to the connector port. To get a more detailed description of the connectors, please download the SmartConnector User Guide from the official ArcSight SmartConnector documentation.

Used resources

ArcSight Logger configuration

ArcSight Logger has logs receiver Wallarm Fluentd logs configured as follows:

  • Logs are received via UDP (Type = UDP Receiver)

  • Listening port is 514

  • Events are parsed with the syslog parser

  • Other default settings

Configuration of receiver in ArcSight Logger

To get a more detailed description of the receiver configuration, please download the Logger Installation Guide of an appropriate version from the official ArcSight Logger documentation.

Fluentd configuration

Fluentd is configured in the td-agent.conf file:

  • Incoming webhook processing is configured in the source directive:

    • All HTTP and HTTPS traffic is sent to 9880 Fluentd port
    • TLS certificate for HTTPS connection is located within the file /etc/pki/ca.pem
  • Forwarding logs to ArcSight Logger and log output are configured in the match directive:

    • All event logs are copied from Fluentd and forwarded to ArcSight Logger at the IP address
    • Logs are forwarded from Fluentd to ArcSight Logger in the JSON format according to the Syslog standard
    • Connection with ArcSight Logger is established via UDP
    • Fluentd logs are additionally printed on the command line in JSON format (19-22 code lines). The setting is used to verify that events are logged via Fluentd
 2  @type http # input plugin for HTTP and HTTPS traffic
 3  port 9880 # port for incoming requests
 4  <transport tls> # certificates for HTTPS connection
 5    ca_path /etc/pki/ca.pem
 6  </transport>
 8<match **>
 9  @type copy
10  <store>
11      @type remote_syslog # output plugin to forward logs from Fluentd via Syslog
12      host # IP address to forward logs to
13      port 514 # port to forward logs to
14      protocol udp # connection protocol
15    <format>
16      @type json # format of forwarded logs
17    </format>
18  </store>
19  <store>
20     @type stdout # output plugin to print Fluentd logs on the command line
21     output_type json # format of logs printed on the command line
22  </store>

A more detailed description of configuration files is available in the official Fluentd documentation.

Testing Fluentd configuration

To check that Fluentd logs are created and forwarded to ArcSight Logger, the PUT or POST request can be sent to Fluentd.

Request example:

curl -X POST '' -H "Content-Type: application/json" -d '{"key1":"value1", "key2":"value2"}'

Fluentd logs:

Logs in Fluentd

Event in ArcSight Logger:

Logs in ArcSight Logger

Configuration of webhook integration

  • Webhooks are sent to

  • Webhooks are sent via POST requests

  • Additional authentication parameter X-Auth-Token is passed in the request

  • Webhooks sent to Webhook URLs are all available events: hits, system events, vulnerabilities, scope changes

Webhook integration with Fluentd

Example testing

To test the configuration, a new user is added in Wallarm Console:

Adding user

Fluentd will log the event as follows:

Fluentd log about new user

The following entry will be displayed in ArcSight Logger events:

Events in ArccSiight Logger