Getting Data In

How can I calculate in real-time the data rate (kbps) of events being indexed?

hexx
Splunk Employee
Splunk Employee

I would like to find out how much data per second (let's say in kilobytes per second) my indexer is receiving and indexing.

What search could help me find this out?

Tags (3)
1 Solution

hexx
Splunk Employee
Splunk Employee

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermic_noodle | eval search_time=now() | eval seconds_elapsed=(time() - search_time) | eval secs=if(seconds_elapsed<0,"1",seconds_elapsed) | eval esize=((len(_raw)/1024)) | stats sum(esize) as sum_esize, last(secs) AS seconds | stats last(sum_esize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sum_esize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

View solution in original post

mattlucas719
Explorer

better:
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/queues
| search title=parsingQueue* OR title=aggQueue* OR title=typingQueue* OR title=indexQueue*
| eval fill_perc=round(current_size_bytes / max_size_bytes * 100,2)
| fields splunk_server, title, fill_perc
| rex field=title "(?<queue_name>^\w+)(?:\.(?<pipeline_number>\d+))?"
| eval fill_perc = if(isnotnull(pipeline_number), "pset".pipeline_number.": ".fill_perc, fill_perc)
| chart values(fill_perc) over splunk_server by queue_name
| eval pset_count = mvcount(parsingQueue)
| join type=outer splunk_server [
| rest splunk_server_group=dmc_group_indexer splunk_server_group="dmc_group_indexer" /services/server/introspection/indexer
| eval average_KBps = round(average_KBps, 0)
| eval status = if((reason == ".") OR (reason == "") OR isnull(reason), status, status.": ".reason)
| fields splunk_server, average_KBps, status]
| fields splunk_server pset_count average_KBps status parsingQueue aggQueue typingQueue indexQueue
| sort -average_KBps
| stats sum(average_KBps) AS "average_KBps"
| eval "average_MBs"=round(average_KBps/1024, 2)

0 Karma

hexx
Splunk Employee
Splunk Employee

Run the following search over the time range real-time(all time) and you will be able to see what the incoming data rate (in kilobytes of rawdata per second) for a given subset of event looks like :

host=hypodermic_noodle | eval search_time=now() | eval seconds_elapsed=(time() - search_time) | eval secs=if(seconds_elapsed<0,"1",seconds_elapsed) | eval esize=((len(_raw)/1024)) | stats sum(esize) as sum_esize, last(secs) AS seconds | stats last(sum_esize) AS "event rawdata indexed (kb)", last(seconds) AS "search seconds elapsed", last(eval(sum_esize/seconds)) AS kbps

This example targets a specific host, but feel free to change the first search terms to better suit your needs.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...