Skip to content

Postanalytics Metrics of the Native Node

This article describes the Postanalytics module and the service runtime metrics of the Native Node to help monitor and troubleshoot the Native Node.

  • The Postanalytics module uses wstore for local traffic processing. Its metrics are prefixed with wallarm_wstore_* and reflect the performance of the Postanalytics module.

    The available metric groups are listed below. The exact list of metrics may vary depending on the Native Node version. Changes are reflected in the Native Node changelog.

  • The wstore service runtime metrics cover network activity, request processing, queue states, storage efficiency, and internal engine health.

Requirements

The Postanalytics module and the service runtime metrics of the Native Node are available starting from version 0.20.0 for all deployment options except Amazon Machine Image (AMI).

Metrics endpoint

By default, the Native Node provides the metrics at the following endpoint using the tcp4 (IPv4-only) protocol:

http://localhost:9001/metrics

This endpoint is accessible only from the server itself.

Security note

Unless you need to publicly expose the metrics endpoint (e.g., to run a Prometheus metrics scraper), we recommend using the default localhost listen address.

You can change the default metrics host and endpoint. Follow the instructions for your deployment type:

For All-in-one installer and Docker image:

Set the following environment variables when deploying the Native Node:

  • WALLARM_WSTORE__METRICS__LISTEN_ADDRESS — specifies the host and port

  • WALLARM_WSTORE__METRICS__PROTOCOL — specifies the protocol

For Helm chart:

Configure the following settings in values.yaml during or after Native Node deployment:

Blocked requests metrics

wallarm_wstore_blocked_stat_ack_missed

The number of acknowledgement attempts for blocked requests that were missed or not processed.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_ack_missed 0

wallarm_wstore_blocked_stat_ack_success

The number of blocked requests successfully acknowledged.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_ack_success 0

wallarm_wstore_blocked_stat_merged_new

The number of new blocked requests merged into existing blocked request records.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_merged_new 0

wallarm_wstore_blocked_stat_merged_updates

The number of updates applied to existing blocked requests when merging new information.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_merged_updates 0

wallarm_wstore_blocked_stat_read_items

The total number of blocked request items read from wstore.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_read_items 0

wallarm_wstore_blocked_stat_register_size

The current size of the blocked requests register.

Type: Gauge

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_register_size 0

wallarm_wstore_blocked_stat_rejected_full

The number of blocked requests rejected due to the blocked requests register being full.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_rejected_full 0

wallarm_wstore_blocked_stat_unacked_size

The number of currently unacknowledged blocked requests.

Type: Gauge

Labels: None

Unit: Count

Example:

wallarm_wstore_blocked_stat_unacked_size 0

Connections and traffic metrics

wallarm_wstore_connections_total

The total number of network connections handled by wstore, broken down by connection type (i.e., protocol schema like TCP or TLS).

Type: Counter

Labels:

  • TCP

  • TLS

Unit: Count

Example:

wallarm_wstore_connections_total{schema="TCP"} 219

wallarm_wstore_current_connections

The number of active connections currently established with wstore.

Type: Gauge

Labels: None

Unit: Count

Example:

wallarm_wstore_current_connections 9

wallarm_wstore_requests_total

The total number of requests processed, labeled by the request code and the result of the operation (success or failed).

Type: Counter

Labels:

  • code - type of the IPROTO request (e.g., IPROTO_CALL, IPROTO_CALL_16, IPROTO_ID, etc.)

  • result - result of the operation (success or failed)

Unit: Count

Example:

wallarm_wstore_requests_total{code="IPROTO_CALL_16",result="success"} 5962210

wallarm_wstore_iproto_calls_total

The total number of iproto CALL/CALL_16 requests, broken down by the function name and result.

Type: Counter

Labels:

  • func - name of the called function

  • result - result of the operation (success or failed)

Unit: Count

Example:

wallarm_wstore_iproto_calls_total{func="wallarm.blocked_stat.read",result="success"} 621473

Request queue metrics

wallarm_wstore_queue_size

The current number of requests in each wstore queue.

Type: Gauge

Labels:

  • engine - e.g., ring

  • name - name of the wstore queue

Unit: Count

Example:

wallarm_wstore_queue_size{engine="ring",name="api_discovery"} 0

wallarm_wstore_queue_drops

The number of requests dropped when a wstore queue reaches its maximum size and begins overwriting entries in the ring buffer, broken down by queue.

Type: Counter

Labels: queue - name of the wstore queue

Unit: Count

Example:

wallarm_wstore_queue_drops{queue="appstructure"} 0

wallarm_wstore_queue_take_requests

The number of requests returned from the queue by the wallarm.requests_processing.take function, labeled by the result of the operation (success or failed).

Type: Counter

Labels:

  • queue - name of the wstore queue

  • result - result of the operation (success or failed)

Unit: Count

Example:

wallarm_wstore_queue_take_requests{queue="appstructure", result="success"} 313187

wallarm_wstore_queue_ack_drops

The number of acknowledgement attempts for requests that have already been removed from the wstore queue.

Type: Counter

Labels: queue - name of the wstore queue

Unit: Count

Example:

wallarm_wstore_queue_ack_drops{queue="appstructure"} 0

wallarm_wstore_queue_ack_return

The number of requests that were captured but not acknowledged, and were therefore returned to the queue for reprocessing.

Type: Counter

Labels:

  • queue - name of the wstore queue

  • result - result of the operation (success or failed)

Unit: Count

Example:

wallarm_wstore_queue_ack_return{queue="appstructure",result="failed"} 0

wallarm_wstore_queue_stats

The total number of put, ack, and take actions per queue, maintained for backward compatibility.

Type: Counter

Labels: none

  • queue - name of the wstore queue

  • action - type of the queue operation (put, take, or ack)

Unit: Count

Example:

wallarm_wstore_queue_stats{queue="appstructure",action="ack"} 770

wallarm_wstore_queue_throttled

The number of requests rejected due to queue throttling, broken down by queue.

Type: Counter

Labels: queue - name of the wstore queue

Unit: Count

Example:

wallarm_wstore_queue_throttled{queue="appstructure"} 0

Request storage metrics

wallarm_wstore_request_storage_total_size

The total size of all stored requests in bytes.

Type: Gauge

Labels: None

Unit: Bytes

Example:

wallarm_wstore_request_storage_total_size 2285568

wallarm_wstore_request_storage_timeframe_size

Current time span in seconds between the oldest and newest requests stored in wstore.

Type: Gauge

Labels: None

Unit: Seconds

Example:

wallarm_wstore_request_storage_timeframe_size 308775

wallarm_wstore_request_storage_drops

The number of old requests dropped to make room for new ones when the maximum request storage size is reached.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_request_storage_drops 0

wallarm_wstore_request_storage_rejects

The number of incoming requests rejected because they are too large to be stored.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_request_storage_rejects 0

Internal wstore engine metrics

wallarm_wstore_kvstore_records_total

The total number of records currently stored in the wstore key-value store.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_kvstore_records_total 770

wallarm_wstore_kvstore_cleanups

The number of old requests cleaned up from the wstore internal key-value store.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_kvstore_cleanups 0

wallarm_wstore_kvstore_errors

The number of errors in the wstore internal key-value store operations, labeled by action type (e.g., cleanup, insert, or drop).

Type: Counter

Labels:

  • cleanup

  • drop

  • get_size

  • insert

Unit: Count

Example:

wallarm_wstore_kvstore_errors{action="cleanup"} 0

wallarm_wstore_kvstore_oom_errors_total

The number of Out Of Memory (OOM) errors occurred during insertion into the wstore key-value store.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_kvstore_oom_errors_total 0

wallarm_wstore_kvstore_insertions_total

The number of requests successfully stored by wstore into its key-value store.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_kvstore_insertions_total 770

wallarm_wstore_kvstore_lost_insertions_total

The number of requests failed to be stored in the wstore key-value store after all retry attempts.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_kvstore_lost_insertions_total 0

wallarm_wstore_kvstore_drops_total

The number of requests lost due to failed cleanups from the wstore internal key-value store.

Type: Counter

Labels: None

Unit: Count

Example:

wallarm_wstore_kvstore_drops_total 0

Service runtime metrics

We can divide service runtime metrics into 2 groups:

  • Prometheus Go metrics, prefixed with go_* and process_*

    Most Prometheus Go metrics are documented here. If a metric is missing there, refer to the official Go metrics documentation.

  • Prometheus metrics, prefixed with metrics_push_*, related to its push gateway.

See the example of service runtime metrics below.

go_sched_latencies_seconds_bucket{le="0"} 0
go_sched_latencies_seconds_bucket{le="2.56e-07"} 5709872
go_sched_latencies_seconds_bucket{le="4.48e-07"} 5852648
go_sched_latencies_seconds_bucket{le="7.68e-07"} 6167836
go_sched_latencies_seconds_bucket{le="1.28e-06"} 6659543
go_sched_latencies_seconds_bucket{le="2.048e-06"} 6712660
go_sched_latencies_seconds_bucket{le="3.584e-06"} 6721120
go_sched_latencies_seconds_bucket{le="6.144e-06"} 6728752
go_sched_latencies_seconds_bucket{le="1.024e-05"} 6770348
go_sched_latencies_seconds_bucket{le="1.6384e-05"} 6864636
go_sched_latencies_seconds_bucket{le="3.2768e-05"} 6945503
go_sched_latencies_seconds_bucket{le="5.7344e-05"} 7023656
go_sched_latencies_seconds_bucket{le="9.8304e-05"} 7058106
go_sched_latencies_seconds_bucket{le="0.00016384"} 7076853
go_sched_latencies_seconds_bucket{le="0.000262144"} 7081681
go_sched_latencies_seconds_bucket{le="0.000458752"} 7083229
go_sched_latencies_seconds_bucket{le="0.000786432"} 7083551
go_sched_latencies_seconds_bucket{le="0.00131072"} 7083749
go_sched_latencies_seconds_bucket{le="0.002097152"} 7083849
go_sched_latencies_seconds_bucket{le="0.003670016"} 7083899
go_sched_latencies_seconds_bucket{le="0.007340032"} 7083926
go_sched_latencies_seconds_bucket{le="0.012582912"} 7083936
go_sched_latencies_seconds_bucket{le="0.02097152"} 7083939
go_sched_latencies_seconds_bucket{le="0.033554432"} 7083940
go_sched_latencies_seconds_bucket{le="0.058720256"} 7083940
go_sched_latencies_seconds_bucket{le="0.100663296"} 7083940
go_sched_latencies_seconds_bucket{le="0.16777216"} 7083940
go_sched_latencies_seconds_bucket{le="0.268435456"} 7083940
go_sched_latencies_seconds_bucket{le="0.469762048"} 7083940
go_sched_latencies_seconds_bucket{le="0.805306368"} 7083940
go_sched_latencies_seconds_bucket{le="+Inf"} 7083940
go_mutex_wait_seconds_total 20.190952976
go_gc_mark_assist_cpu_seconds_total 1.461018963
go_gc_cpu_seconds_total 93.183776342
go_gc_pauses_seconds_bucket{le="0"} 0
go_gc_pauses_seconds_bucket{le="2.56e-07"} 0
go_gc_pauses_seconds_bucket{le="4.48e-07"} 0
go_gc_pauses_seconds_bucket{le="7.68e-07"} 0
go_gc_pauses_seconds_bucket{le="1.28e-06"} 0
go_gc_pauses_seconds_bucket{le="2.048e-06"} 0
go_gc_pauses_seconds_bucket{le="3.584e-06"} 0
go_gc_pauses_seconds_bucket{le="6.144e-06"} 5
go_gc_pauses_seconds_bucket{le="1.024e-05"} 78
go_gc_pauses_seconds_bucket{le="1.6384e-05"} 5704
go_gc_pauses_seconds_bucket{le="3.2768e-05"} 6651
go_gc_pauses_seconds_bucket{le="5.7344e-05"} 10511
go_gc_pauses_seconds_bucket{le="9.8304e-05"} 12303
go_gc_pauses_seconds_bucket{le="0.00016384"} 13390
go_gc_pauses_seconds_bucket{le="0.000262144"} 13841
go_gc_pauses_seconds_bucket{le="0.000458752"} 13974
go_gc_pauses_seconds_bucket{le="0.000786432"} 14006
go_gc_pauses_seconds_bucket{le="0.00131072"} 14020
go_gc_pauses_seconds_bucket{le="0.002097152"} 14025
go_gc_pauses_seconds_bucket{le="0.003670016"} 14031
go_gc_pauses_seconds_bucket{le="0.007340032"} 14040
go_gc_pauses_seconds_bucket{le="0.012582912"} 14041
go_gc_pauses_seconds_bucket{le="0.02097152"} 14044
go_gc_pauses_seconds_bucket{le="0.033554432"} 14044
go_gc_pauses_seconds_bucket{le="0.058720256"} 14044
go_gc_pauses_seconds_bucket{le="0.100663296"} 14044
go_gc_pauses_seconds_bucket{le="0.16777216"} 14044
go_gc_pauses_seconds_bucket{le="0.268435456"} 14044
go_gc_pauses_seconds_bucket{le="0.469762048"} 14044
go_gc_pauses_seconds_bucket{le="0.805306368"} 14044
go_gc_pauses_seconds_bucket{le="+Inf"} 14044
go_scavenge_cpu_seconds_total 0.770850166
go_memlimit_bytes 268435474
go_memstats_alloc_bytes 6909704
go_memstats_alloc_bytes_total 26189017904
go_memstats_buck_hash_sys_bytes 1687394
go_memstats_frees_total 331930366
go_memstats_gc_cpu_fraction 2.9248506362442843e-05
go_memstats_gc_sys_bytes 3411728
go_memstats_heap_alloc_bytes 6909704
go_memstats_heap_idle_bytes 7069696
go_memstats_heap_inuse_bytes 8888320
go_memstats_heap_objects 155526
go_memstats_heap_released_bytes 6832128
go_memstats_heap_sys_bytes 15958016
go_memstats_last_gc_time_seconds 1.7645698917087147e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 332085892
go_memstats_mcache_inuse_bytes 4832
go_memstats_mcache_sys_bytes 15704
go_memstats_mspan_inuse_bytes 202240
go_memstats_mspan_sys_bytes 261120
go_memstats_next_gc_bytes 11218594
go_memstats_other_sys_bytes 741694
go_memstats_stack_inuse_bytes 819200
go_memstats_stack_sys_bytes 819200
go_memstats_sys_bytes 22894856
go_cgo_calls_count 10960395
go_cpu_count 4
go_gc_duration_seconds{quantile="0"} 5.1574e-05
go_gc_duration_seconds{quantile="0.25"} 6.0257e-05
go_gc_duration_seconds{quantile="0.5"} 7.6857e-05
go_gc_duration_seconds{quantile="0.75"} 0.000138767
go_gc_duration_seconds{quantile="1"} 0.00222732
go_gc_duration_seconds_sum 0.84562571
go_gc_duration_seconds_count 7022
go_gc_forced_count 1726
go_gomaxprocs 4
go_goroutines 29
go_threads 13
go_info{version="go1.25.4"} 1
go_info_ext{compiler="gc", GOARCH="amd64", GOOS="linux", GOROOT="/usr/local/go"} 1
process_cpu_seconds_system_total 1061.14
process_cpu_seconds_total 2303.44
process_cpu_seconds_user_total 1242.3
process_major_pagefaults_total 0
process_minor_pagefaults_total 1544282
process_num_threads 12
process_resident_memory_bytes 26955776
process_start_time_seconds 1764310991
process_virtual_memory_bytes 1273065472
process_virtual_memory_peak_bytes 1273098240
process_resident_memory_peak_bytes 30531584
process_resident_memory_anon_bytes 16076800
process_resident_memory_file_bytes 10878976
process_resident_memory_shared_bytes 0
process_io_read_bytes_total 528866931
process_io_written_bytes_total 274111565
process_io_read_syscalls_total 18250793
process_io_write_syscalls_total 14300084
process_io_storage_read_bytes_total 0
process_io_storage_written_bytes_total 0
metrics_push_block_size_bytes_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="1.136e+04...1.292e+04"} 2
metrics_push_block_size_bytes_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="1.292e+04...1.468e+04"} 8628
metrics_push_block_size_bytes_sum{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 112724778
metrics_push_block_size_bytes_count{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 8630
metrics_push_bytes_pushed_total{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 112724778
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="1.668e-01...1.896e-01"} 28
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="1.896e-01...2.154e-01"} 7561
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="2.154e-01...2.448e-01"} 746
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="2.448e-01...2.783e-01"} 107
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="2.783e-01...3.162e-01"} 46
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="3.162e-01...3.594e-01"} 30
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="3.594e-01...4.084e-01"} 81
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="4.084e-01...4.642e-01"} 10
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="4.642e-01...5.275e-01"} 9
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="5.275e-01...5.995e-01"} 3
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="5.995e-01...6.813e-01"} 3
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="6.813e-01...7.743e-01"} 1
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="7.743e-01...8.799e-01"} 2
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="1.292e+00...1.468e+00"} 2
metrics_push_duration_seconds_bucket{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics",vmrange="1.668e+00...1.896e+00"} 1
metrics_push_duration_seconds_sum{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 1801.3603247859955
metrics_push_duration_seconds_count{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 8630
metrics_push_errors_total{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 0
metrics_push_interval_seconds{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 30
metrics_push_total{url="https://api.wallarm.com:443/v2/node/stat/victoriametrics"} 8630