Over the holiday weekend our license usage doubled and I need to figure out what is causing this. I have S.o.S - Splunk on Splunk installed but I am not sure if this is where I find this information. anyone have any searches to determine what devices started sending data or volume change?
Try running these searches for the last 10 days:
| tstats count GroupBy index _time span=1d
| timechart span=1d sum(count) BY index
Get more granular with this:
| tstats count GroupBy sourcetype index _time span=1d
| eval entity = sourcetype . "/" . index
| timechart span=1d sum(count) BY entity
Get the most granularity with this:
| tstats count GroupBy host sourcetype index _time span=1d
| eval entity = host . "/" . sourcetype . "/" . index
| timechart span=1d sum(count) BY entity
These will chart changes in # of events, but if the problem is that this metric is more or less the same and the problem is that the average size of a particular event has increased, then it will not be evident.
Try running these searches for the last 10 days:
| tstats count GroupBy index _time span=1d
| timechart span=1d sum(count) BY index
Get more granular with this:
| tstats count GroupBy sourcetype index _time span=1d
| eval entity = sourcetype . "/" . index
| timechart span=1d sum(count) BY entity
Get the most granularity with this:
| tstats count GroupBy host sourcetype index _time span=1d
| eval entity = host . "/" . sourcetype . "/" . index
| timechart span=1d sum(count) BY entity
These will chart changes in # of events, but if the problem is that this metric is more or less the same and the problem is that the average size of a particular event has increased, then it will not be evident.
Thank you! this helped determine what index doubled in size. How do i run these searches for that particular index? I tried adding index=XXX but the search did not like it at all
Also, check out the Meta Woot!
app:
Oh this looks like a great app!!! do i have to install it in all HF and indexers? I installed in the search head and no data came up.
After the tstats
line add this line:
| search index="XXX"
NICE! that worked!!