Getting Data In

How to reduce the daily ingestion on Splunk cloud?

VinodTiwari
New Member

Hi,

We use splunk cloud and our daily ingestion limit is 800 GB, we are ingesting about 100 GB over the limit. I'm looking for suggestions as in what can be done to reduce the ingestion. I did some digging and found we were sending debug logs which we didn't need, but that is just 10 GB per day. I'm fairly new to splunk and seeking help in understanding how this can be achieved.

Thanks

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the inputs.conf files on your forwarders. Your monitor:// stanzas may be looking at more files than they need to. If you rotate your log files, make sure you are not ingesting the rolled logs ("foo.log.1", for example).

---
If this reply helps you, Karma would be appreciated.
0 Karma

skoelpin
SplunkTrust
SplunkTrust

I've seen this happen at a customer site. They were monitoring everything with a * and when the logs rolled, the fishbucket got confused and thought the rolled log was a new file not seen before and massively increased volumes. The solution was to monitor the active log file and to use the crcSalt = <SOURCE> attribute to add more uniqueness to the hash

skoelpin
SplunkTrust
SplunkTrust

Hi @VinodTiwari, welcome to Answers!

How many servers and files are you currently monitoring? I would recommend leaving only the mission critical stuff logging first and turn everything else off. You should then find out what you don't need to monitor and exclude it.

You most likely have a * in your inputs.conf to monitor all the files, so you should remove that and write an expression to capture the files you want to ingest

You should also look into turning off debug mode if not needed and also use nullQueue to exclude certain text from logs that you don't need

https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Getstartedwithgettingdatain

https://answers.splunk.com/answers/305311/how-to-send-events-to-the-nullqueue-on-indexer.html

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...