Getting Data In

monitor config for log files - universal forwarder

lakshman237
Path Finder

I have log files, say, "logFile1.txt", "logFile2.txt" in folder /home/system/logs/ . The folder also has rotated logs which are of the form "logFile1.201206021010.txt" ( yyyymmddhhmm) added.
[monitor:////home/system/logs/logFile*.txt]
disabled = false
sourcetype = mysystem
index = myindex

The above config brings rotated logs well to the index and sourcetype, which I donot want. I can add two stanza's one for logFile1.txt and another for logFile2.txt. However is there a better way to do this?

Tags (3)
0 Karma

Ayn
Legend

This docs section covers how Splunk handles rotated files. Essentially, when you initially add a directory for monitoring Splunk will read all of the files in there because it hasn't seen any of them before, but after that it will never re-index a rotated file because the contents will be the same as before.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

0 Karma

sdaniels
Splunk Employee
Splunk Employee

You probably want to add an entry to inputs.conf for crcSalt.

crcSalt=<\SOURCE>

crcSalt = /
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only
performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same
file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the
CRC is based on only the first few lines of the file, it is possible for legitimately different files to have
matching CRCs, particularly if they have identical headers.)
* If set, is added to the CRC.
* If set to the literal string (including the angle brackets), the full directory path to the source file
is added to the CRC. This ensures that each file being monitored has a unique CRC. When crcSalt is invoked,
it is usually set to .
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed
after it has rolled.
* Defaults to empty.

0 Karma

sdaniels
Splunk Employee
Splunk Employee

Let me know what you find out. Logs with very large headers cause problems since Splunk doesn't detect any change. This is addressed in our next major version.

0 Karma

lakshman237
Path Finder

thanks, but will this ensure logFile1.txt and LogFile2.txt are indexed but not the rotated files? ( with the above monitor command). I had an issue with double indexing in the past with crcSalt. let me check this again.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...