Getting Data In

Splunk occasionally not capturing all logs

rileykohl21
New Member

Hey all, I'm running into some odd behavior. I currently have splunkforwarder set up on a container and it should be pulling in logs from three log files. Here's a quick rundown of the issue:

  • "daily.one.log", "daily.two.log", and "daily.three.log" are generated when I run three different backups for another application
  • Backups run around 1am every morning and are usually pretty quick (a matter of minutes)
  • a Splunk search for those files usually returns the complete contents of all three logs for a given day
  • every few days or so, Splunk will not have all of the lines from the log files or even all the logs
  • A recent example would be that Splunk had the complete logs for "daily.one.log", nothing from "daily.two.log", and some of the lines from "daily.three.log"

Any ideas as to what could be causing this to happen sporadically?

Tags (2)
0 Karma

whrg
Motivator

Perhaps you can find the source of the problem via Add Data / Upload in Splunk web interface: Upload a faulty file, select the according sourcetype and check if everything is alright.

One idea: Perhaps timestamp recognition is not working correctly. Again, use Add Data / Upload to verify that all timestamps are parsed correctly. If not, have a look at Configure timestamp recognition, specifically TIME_FORMAT and TIME_PREFIX in props.conf.

Another idea: Splunk will not index files which are completely identical so that Splunk does not index log rotated files. (I'm not sure though if this applies in your case, since you said that sometimes only partial files are indexed.)
You can prevent this behaviour in your inputs.conf with:

crcSalt = <SOURCE>
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...