Splunk Search

How to check the universal forwarder's metrics.log to get instantaneous_kbps and average_kbps?

kteng2024
Path Finder

From Documentation:

To verify how often the forwarder is hitting this limit, check the forwarder's metrics.log. (Look for this on the forwarder because metrics.log is not forwarded by default on universal and light forwarders.)

cd $SPLUNK_HOME/var/log/splunk/metrics.log
grep "name=thruput" metrics.log

Example: The instantaneous_kbps and average_kbps are always under 256KBps.

11-19-2013 07:36:01.398 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=251.790673,instantaneous_eps=3.934229, average_kbps=110.691774, total_k_processed=101429722, kb=7808.000000, ev=122

But when i run the grep -i "name=thruput" metrics.log , i wouldn't get any result. So, can i please know whether there is any way to check the instantaneous_kbps and average_kbps ?

0 Karma

woodcock
Esteemed Legend

Try this:
|tstats avg(tcp_KBps) AS "KBps" avg(tcp_eps) AS "Eps"
WHERE index=_internal AND sourcetype="splunkd" AND connectionType IN("cooked", "cookedSSL")
AND hostname="*"
BY hostname _time [| makeresults count=2
| streamstats count
| eval _time = _time - (100 * count)
| timechart count
| rename _* AS INV_*
| eval search = "span=" . INV_span . "s" ]
| rename hostname AS host
| timechart limit=0 useother=f first(KBps) AS KBps first(Eps) AS Eps BY host

0 Karma

woodcock
Esteemed Legend

Why bother going to the forwarder, just do this:

index=_internal source="*metrics.log" component="Metrics" host=MyHost
0 Karma

GregZillgitt
Path Finder

Can you load metrics.log into vi and search there?

I just ssh'd into one of our forwarders and it worked for me...

$ grep -i "name=thruput" metrics.log
01-26-2017 15:16:23.841 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.202834, instantaneous_eps=0.257975, average_kbps=19.430535, total_k_processed=4248380.000000, kb=6.290039, ev=8.000000, load_average=2.010000
01-26-2017 15:16:54.829 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.219149, instantaneous_eps=0.613138, average_kbps=19.427809, total_k_processed=4248386.000000, kb=6.791016, ev=19.000000, load_average=1.610000
01-26-2017 15:17:25.829 -0600 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.209961, instantaneous_eps=0.258064, average_kbps=19.425083, total_k_processed=4248392.000000, kb=6.508789, ev=8.000000, load_average=1.500000
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...