Getting Data In

duplicated events when monitoring from log file

remy06
Contributor

Hi,

I have a script that pulls oracle events and write them to a file called ora.log.The script runs at 5 min interval.

After which I've configured splunk to monitor that file as data input.Currently I've noticed there are some duplication of events when i do a search..

How do I configure splunk to indexed only the new events after subsequent runs?

1 Solution

simuvid
Splunk Employee
Splunk Employee

Hi there,

when you integrate or uploaded logfiles from a directory as a new data input, you can specify a setting for this data input via the Splunk Management UI.

Set the flag for the setting Follow Tail.

When you want to modify this setting in the inputs.conf file, just add following line to the file:

followTail = 1

That tells Splunk only to read out the new events from logfiles.

Hope that's what you are looking for!

Cheers,

Christian

View solution in original post

simuvid
Splunk Employee
Splunk Employee

Hi remy06,

the ora.log file has it something like a timestamp in the filename, or something else that changes the filename after adding new entries into it?
Or do you have any kind of header in the logfile?

0 Karma

chjpcert
Explorer

I've got the same problem here. I download a log file to a temporary directory every 5 minutes, then move it into the log file directory I've specified in Splunk, overwriting the previous log.

However, many events are duplicated in the index. For example, one log with 8859 lines ended up as 154,130 events in the index. Adding it manually via the "add oneshot" command produces the correct number of events.

I've confirmed that the events are listed only once in the log files themselves. I've got followTail = 1 set in inputs.conf. I've also got crcSalt = set, if that's somehow related.

Is there something off with the way Splunk handles tailing log files, or is there a config change needed here?

MousumiChowdhur
Contributor

Hi!!

Does "followTail" work in case of windows logs? If so, do I have to have crcSalt set to some text alongwith followTail?

Thanks!

0 Karma

remy06
Contributor

some events are being duplicated,not the entire file as I've taken a look at the actual ora.log file.

0 Karma

Jeremiah
Motivator

Is it really only that some events that are duplicated, or is the entire file getting re-indexed each time the script updates the log?

simuvid
Splunk Employee
Splunk Employee

Hi there,

when you integrate or uploaded logfiles from a directory as a new data input, you can specify a setting for this data input via the Splunk Management UI.

Set the flag for the setting Follow Tail.

When you want to modify this setting in the inputs.conf file, just add following line to the file:

followTail = 1

That tells Splunk only to read out the new events from logfiles.

Hope that's what you are looking for!

Cheers,

Christian

remy06
Contributor

Am not sure if its related to known issue (SPL-23555) where "monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended." ?.....

0 Karma

remy06
Contributor

I've just noticed that when enabled tailing, some events are truncated off..

0 Karma

remy06
Contributor

I've tried.Frequency of duplicate events seem to have reduced a little, but still the problem exist. Is it a bug?Or a configuration issue?

Also by using this method,earlier events in the file does not get indexed as monitoring starts at the end of the file..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...