Why are audittrail and splunkd sourcetypes eating up 1/5th of my allowed indexing?

[juriggs]

So I was confused as to why the small amount I was indexing from my event logs every day was getting me so close to my 500MB limit... when I looked at the stats, the audittrail sourcetype was indexing 70MB a day, and the splunkd sourcetype was logging 30MB a day. Uh, what? 1/5th of my allowed indexing per day has nothing to do with anything I care about?

How do I disable these sourcetypes, and what are the consequences of doing so?

Thanks...

[Drainy]

I'm a little confused by the question and the answer. Even if Splunk went mad and re-indexed everything it is all fed into _audit and _internal which doesn't count towards your license usage. Have you verified which index it is feeding into? I'm wondering if the input is being duplicated somewhere or something has gone wrong with an upgrade perhaps?

Also if its reading as audittrail then this suggests its Splunks own generated audit logs which are then fed into the audit.log, the actual audit index appears to be filled by Splunk itself. (I've had some issues with this index for security reasons and have had to blacklist and reindex and it appears with duplicates, again it doesn't count towards my license as its feeding into an internal index

[grijhwani]

The internal logs DO contribute toward your licence usage. At the very least the throughput is logged in the audits files, and contributes to the usage figure returned by a query I was given by Splunk support for the Enterprise installation at work.

In an Enterprise installation the Splunk logs are just a drop in the ocean so go unnoticed. In the free installation they certainly seem to get recorded as contrbuting to licence consumption.

[lguinn2]

Sorry, you are just wrong about this. The quantities may show up in various reports but I assure you that they do not count against your license.

[grijhwani]

I had the same problem. It is because Splunk is rotating its own log files and retaining them for 5 days, but it doesn't come with a blacklist to automatically suppress the rotated logs. Nett result: all your Splunk logs are being re-indexed at midnight, ultimately 6 times each (original file and 5 daily rotations).
I have a nice little search which demonstrates the effect:

index="_internal" source="*license_usage.log" | eval ISODate=strftime(strptime(date_year."-".date_month."-".date_mday, "%Y-%b-%d"), "%Y-%m-%d (%a)") | eval kB=b/1024 | chart eval(round(sum(kB),0)) over date_hour by ISODate limit=0  | addcoltotals labelfield=date_hour

You if you activate the "heat map" presentation you will probably instantly see a red blip just after midnight every day.

You don't really want to suppress them completely if you want to learn anything about Splunk, and they are your ultimate source of information if you ever come to diagnose problems. (Not even sure that they can be suppressed.) You can reduce the effect, though: put the following in $SPLUNK_HOME/etc/system/local/inputs.conf in the [defaults] stanza (creating it at the top of the file if need be):

blacklist = .*\.([0-9]+|gz|bz2)$

Then restart Splunk.

You will now have suppressed any log files with sequence numbers or that have previously been compressed. Obviously this does not work retrospectively, but from here on you should only have your live daily splunk logs indexed, and any rotations will be ignored. This has the benefit of applying the same default blacklist across the board.

It is possible to see which input sources are being consumed and which not, in real time, but only by tinkering with your access configs first.

The URL is:

https://{yoursplunkserver}:8089/services/admin/inputstatus/TailingProcessor:FileStatus

It will tell you what needs changing to make it work.

[grijhwani]

And you could do me a favour by clicking the "accepted answer".

[grijhwani]

I have made a correction to the blacklist regex.

[juriggs]

this seems to have worked... it only indexed 10MB or so yesterday. Thanks!

[grijhwani]

And yes, I am aware that I should be using "convert" but when I first created this search I was a novice (and in practical terms, I still am).