Datadog via Fluentd/Logstash¶

You can set up Wallarm to send notifications of detected events to Datadog through the Fluentd or Logstash intermediate data collector.

The most common logging scheme in complex systems consists of the following components:

• Data collector: accepts logs from several sources and forwards logs to the SIEM system

• SIEM system or log management systems: used to analyze logs and monitor the system status

Used resources¶

• The Fluentd or Logstash service available on the public URL

• The Datadog service available on the public URL

• Administrator access to Wallarm Console in EU cloud to configure the Fluentd/Logstash integration

34.94.85.217
34.94.51.234
34.102.59.122
34.94.238.72

34.141.230.156
34.91.138.113
34.90.114.134
35.204.127.78

Requirements¶

Since Wallarm sends logs to the intermediate data collector via webhooks, the configuration of Fluentd or Logstash should meet the following requirements:

• Accept the POST or PUT requests

• Accept HTTPS requests

• Have public URL

• Forward logs to Datadog via the datadog_logs Logstash plugin or the fluent-plugin-datadog Fluentd plugin

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

input {
  http { # input plugin for HTTP and HTTPS traffic
    port => 5044 # port for incoming requests
    ssl => true # HTTPS traffic processing
    ssl_certificate => "/etc/server.crt" # Logstash TLS certificate
    ssl_key => "/etc/server.key" # private key for TLS certificate
  }
}
filter {
  mutate {
    add_field => {
        "ddsource" => "wallarm" # mutate filter adding the source field to the Datadog log record for further filtration of Wallarm logs
    }
  }
}
output {
  stdout {} # output plugin to print Logstash logs on the command line
  datadog_logs { # output plugin to forward the Logstash logs to Datadog
      api_key => "XXXX" # API key generated for the organization in Datadog
      host => "http-intake.logs.datadoghq.eu" # Datadog endpoint (depends on the registration region)
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<source>
  @type http # input plugin for HTTP and HTTPS traffic
  port 9880 # port for incoming requests
  <transport tls> # configuration for connections handling
    cert_path /etc/ssl/certs/fluentd.crt
    private_key_path /etc/ssl/private/fluentd.key
  </transport>
</source>
<match datadog.**>
  @type datadog # output plugin to forward logs from Fluentd to Datadog
  @id awesome_agent
  api_key XXXX # API key generated for the organization in Datadog
  host 'http-intake.logs.datadoghq.eu' # Datadog endpoint (depends on the registration region)

  # Optional
  include_tag_key true
  tag_key 'tag'

  # Optional tags
  dd_source 'wallarm' # adding the source field to the Datadog log record for further filtration of Wallarm logs
  dd_tags 'integration:fluentd'

  <buffer>
          @type memory
          flush_thread_count 4
          flush_interval 3s
          chunk_limit_size 5m
          chunk_limit_records 500
  </buffer>
</match>

Setting up integration with Fluentd or Logstash¶

1. Proceed to the Datadog integration setup in Wallarm Console → Integrations → Fluentd/Logstash.

2. Input the integration name.

3. Specify target Fluentd or Logstash URL (Webhook URL).

4. If required, configure advanced settings:

   • Request method: POST or PUT. By default, POST requests are sent.
   • Request header and its value if the server requires a non-standard header to execute the request. The number of headers is not limited.
   • CA certificate: certificate of CA that signed a server certificate. If CA is publicly trusted, this field is optional. If a server certificate is self-signed, this field is required and must contain a certificate of your own CA that signed a server certificate.
   • Verify TLS certificate: this setting allows to disable verification of the specified server certificate. By default, Wallarm verifies whether a server certificate is signed by a publicly trusted CA. We do not recommend disabling the verification of production server certificates. If your server uses a self-signed TLS certificate, you can add a self-signed CA certificate to allow sending requests to this server.
   • Request timeout, in seconds: if the server does not respond to the request within the specified time, the request fails. By default: 15 seconds.
   • Connection timeout, in seconds: if the connection to the server cannot be established during the specified time, the request fails. By default: 20 seconds.
   • Wallarm data format: either a JSON Array (default) or a New Line Delimited JSON (NDJSON).

5. Choose event types to trigger sending notifications to the specified URL. If the events are not chosen, then notifications will not be sent.

6. Test the integration and make sure the settings are correct.

7. Click Add integration.

Fluentd integration example:

Testing integration¶

Integration testing allows checking configuration correctness, availability of the Wallarm Cloud, and the notification format. To test the integration, you can use the button Test integration when creating or editing the integration.

The integration is tested as follows:

• Test notifications with the prefix [Test message] are sent to the selected system.

• Test notifications cover the following events (each in a single record):

  • New user in the company account
  • Newly detected hit
  • Newly discovered IP in the company scope
  • New trigger in the company account
  • Newly discovered security vulnerability

• Test notifications include test data.

The test log in the Fluentd or Logstash intermediate data collector:

[
    {
        summary:"[Test message] [Test partner(US)] New vulnerability detected",
        description:"Notification type: vuln

                    New vulnerability was detected in your system.

                    ID: 
                    Title: Test
                    Domain: example.com
                    Path: 
                    Method: 
                    Discovered by: 
                    Parameter: 
                    Type: Info
                    Threat: Medium

                    More details: https://us1.my.wallarm.com/object/555


                    Client: TestCompany
                    Cloud: US
                    ",
        details:{
            client_name:"TestCompany",
            cloud:"US",
            notification_type:"vuln",
            vuln_link:"https://us1.my.wallarm.com/object/555",
            vuln:{
                domain:"example.com",
                id:null,
                method:null,
                parameter:null,
                path:null,
                title:"Test",
                discovered_by:null,
                threat:"Medium",
                type:"Info"
            }
        }
    }
]

The test Datadog log:

To find the Wallarm logs among other records, you can use the source:wallarm_cloud search tag in the Datadog Logs service.

Was this page helpful?

How can we improve this article? (Optional)

Information is unclear or confusing

Insufficient information

Outdated or incorrect information

0/240

Send