Hi,
I have a script that pulls oracle events and write them to a file called ora.log.The script runs at 5 min interval.
After which I've configured splunk to monitor that file as data input.Currently I've noticed there are some duplication of events when i do a search..
How do I configure splunk to indexed only the new events after subsequent runs?
Hi there,
when you integrate or uploaded logfiles from a directory as a new data input, you can specify a setting for this data input via the Splunk Management UI.
Set the flag for the setting Follow Tail.
When you want to modify this setting in the inputs.conf file, just add following line to the file:
followTail = 1
That tells Splunk only to read out the new events from logfiles.
Hope that's what you are looking for!
Cheers,
Christian
Hi remy06,
the ora.log file has it something like a timestamp in the filename, or something else that changes the filename after adding new entries into it?
Or do you have any kind of header in the logfile?
I've got the same problem here. I download a log file to a temporary directory every 5 minutes, then move it into the log file directory I've specified in Splunk, overwriting the previous log.
However, many events are duplicated in the index. For example, one log with 8859 lines ended up as 154,130 events in the index. Adding it manually via the "add oneshot" command produces the correct number of events.
I've confirmed that the events are listed only once in the log files themselves. I've got followTail = 1 set in inputs.conf. I've also got crcSalt = set, if that's somehow related.
Is there something off with the way Splunk handles tailing log files, or is there a config change needed here?
Hi!!
Does "followTail" work in case of windows logs? If so, do I have to have crcSalt set to some text alongwith followTail?
Thanks!
some events are being duplicated,not the entire file as I've taken a look at the actual ora.log file.
Is it really only that some events that are duplicated, or is the entire file getting re-indexed each time the script updates the log?
Hi there,
when you integrate or uploaded logfiles from a directory as a new data input, you can specify a setting for this data input via the Splunk Management UI.
Set the flag for the setting Follow Tail.
When you want to modify this setting in the inputs.conf file, just add following line to the file:
followTail = 1
That tells Splunk only to read out the new events from logfiles.
Hope that's what you are looking for!
Cheers,
Christian
Am not sure if its related to known issue (SPL-23555) where "monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended." ?.....
I've just noticed that when enabled tailing, some events are truncated off..
I've tried.Frequency of duplicate events seem to have reduced a little, but still the problem exist. Is it a bug?Or a configuration issue?
Also by using this method,earlier events in the file does not get indexed as monitoring starts at the end of the file..