Revisiting Splunk

[ASW3382]

I am revisiting splunk to see if it will meet our goals. Right now I am working on the initial index of our data gathered via WMI. A problem I am having is figuring out which license we are going to need when we are ready to buy. It seems difficult to view our indexing volume. Right now it appears that our max per day is 2.5GB but I am guessing that is just because we imported a bunch in one day. I tried running index=_internal todaysBytesIndexed LicenseManager-Audit NOT source=*web_service.log | eval Daily_Indexing_Volume_in_MBs = todaysBytesIndexed/1024/1024 | timechart avg(Daily_Indexing_Volume_in_MBs) by host but that doesnt seem to give much data back. I am only monitoring a couple servers right now and the licensing section reports a violation at around 12:02 every morning.
Does anyone have any helpful searches or reports they can share with me that will give a good breakdown of daily usage? Will my usage quiet down after a month or two?

[gkanapathy]

Internal metrics queries will tell you how much was actually indexed during a particular period. If you indexed several days worth of data in one day, this won't be that useful to you, although of course if you know how many days worth of data you indexed, you can get an average. If over the long term you plan to index data in real time and want to find out how much data is timestamped per day after it has been indexed, you can be fairly close with:

* | eval l=len(_raw) | timechart span=1d sum(l)

But be warned that this will take a very long time to run depending on the amount of data in your index and the speed of your hardware, and furthermore this level of precision is not any more useful than averages, so metrics queries are preferred if you can possibly use those.

[gkanapathy]

It would help to know what you mean by the query not giving you much data back. Basically, it should display the daily audit log entries (which are only generated once per day), and the rest of the query just charts it. Is that what you are seeing?

[marcelofinki]

Would you like to try these ?

(FIRST QUERY)

index=_internal todaysBytesIndexed LicenseManager-Audit NOT source=*web_service.log | eval Daily_Indexing_Volume_in_MBs = todaysBytesIndexed/1024/1024 | timechart avg(Daily_Indexing_Volume_in_MBs) by host

(SECOND QUERY)

index=_internal metrics kb series!=_* "group=per_index_thruput" monthsago=1| eval indexed_mb = kb / 1024 |  timechart  fixedrange=t span=1d sum(indexed_mb) | rename sum(indexed_mb) as totalmb

HTH,

Marcelo

[marcelofinki]

(sorry. i am having a hard time trying to get this comment properly posted.
I need to find out how to publish the query with underscores and other chars that are taken as "escape' characters by the interface )
:-P