Getting Data In

Our daily log indexing rate suddenly increased. How do I find out which index is collecting these logs?

kcooper
Communicator

Recently, the ingest rate of logs (GB per day) has tripled on our Splunk server. We are trying to find out what caused the increase in logs per index.

Any help is appreciated
thx

0 Karma
1 Solution

masonmorales
Influencer
0 Karma

masonmorales
Influencer
0 Karma

lguinn2
Legend

My first suggestion is to use the Distributed Management Console that is built into Splunk.

But you can also run this search:

index=_internal source=*metrics.log group=per_index_thruput |
 timechart span=1h sum(kb) as kb_indexed by series 
| rename series as index 

This search will help you identify the most active forwarders in your environment:

index=_internal source=*metrics.log group=tcpin_connections
| eval sourceHost=coalesce(sourceHost,hostname)
| fields sourceHost kb 
| timechart sum(kb) AS kb_forwarded by sourceHost

jeremiahc4
Builder

If you are on Splunk 6.0 or higher you can go to the following site, click Previous 30 days, then split by index, host, sourcetype, etc... to figure out what's sending all that.

http://yourservername:8000/en-US/manager/search/licenseusage

If you don't have access to it, you have to search against your _internal to figure it out which can be a bit trickier, but as a start, the code behind the by-index split on the above page is shown below;

index=_internal source=*license_usage.log type="Usage" | eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) | eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | bin _time span=1d | stats sum(b) as b by _time, pool, s, st, h, idx   | timechart span=1d sum(b) AS volumeB by idx fixedrange=false | foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...