Hi,
We use splunk cloud and our daily ingestion limit is 800 GB, we are ingesting about 100 GB over the limit. I'm looking for suggestions as in what can be done to reduce the ingestion. I did some digging and found we were sending debug logs which we didn't need, but that is just 10 GB per day. I'm fairly new to splunk and seeking help in understanding how this can be achieved.
Thanks
Check the inputs.conf files on your forwarders. Your monitor://
stanzas may be looking at more files than they need to. If you rotate your log files, make sure you are not ingesting the rolled logs ("foo.log.1", for example).
I've seen this happen at a customer site. They were monitoring everything with a * and when the logs rolled, the fishbucket got confused and thought the rolled log was a new file not seen before and massively increased volumes. The solution was to monitor the active log file and to use the crcSalt = <SOURCE>
attribute to add more uniqueness to the hash
Hi @VinodTiwari, welcome to Answers!
How many servers and files are you currently monitoring? I would recommend leaving only the mission critical stuff logging first and turn everything else off. You should then find out what you don't need to monitor and exclude it.
You most likely have a *
in your inputs.conf
to monitor all the files, so you should remove that and write an expression to capture the files you want to ingest
You should also look into turning off debug mode if not needed and also use nullQueue to exclude certain text from logs that you don't need
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Getstartedwithgettingdatain
https://answers.splunk.com/answers/305311/how-to-send-events-to-the-nullqueue-on-indexer.html