Conditionally monitor log files

[nvonkorff]

Background: Active and Standby server with key directories replicated periodically (every 5 mins) via rsync, including shell scripts and logs. The active server syncs changes to standby server. All scheduled scripts check for the existence of a 'live server flag file' e.g. /opt/LIVESERVER.txt, on the local filesystem and will not execute if the file does not exist. This way, crontab can be enabled on both systems but scripts will only execute on the live side, where the LIVERSERVER flag file exists.

I have a Splunk Universal Forwarder installed on each node. Any scripted inputs can simply have the flag file logic added to them, however I am struggling to work out a way that I can conditionally monitor a log file, i.e. only monitor if some file exists on the local filesystem.

At the moment, we manually stop the Splunk Forwarder on the standby side and only ever run the forwarder on the active node. Ideally, I want both forwarders running all the time, so that I can monitor other files/services on both sides, but I don't want to have the rsynced log files read at the live side, the re-read at the standby side. I only ever want the forwarder to monitor log files on the active node.

I know that a "monitor_if_exists" option doesn't exist, but something like below would be exactly what I am after:

[monitor:///apps/log/test.log]
index = test
sourcetype = TestLog
monitor_if_exists = /opt/LIVESERVER.txt

Anyone know of any way to achieve this?

[MHibbin]

So that you retain your duplicated logs, can't you just have another script in place that will check for the liveserver.txt file and only if it exists then rsync your files to another directory on the local server (e.g. "splunkMonFiles")... then it will still only update the files when changes are made (i.e. via rsync), but Splunk will effectively only be reading from the "liveserver" as there will be no changes to the "splunkMonFiles" directory on the standby-server.

Then the only issue would be disk-usage, however if they are just standard text files (etc) then this should not take up too much space.

Hope that makes sense.

MHibbin

[kristian_kolb · Accepted solution]

Hm,

I believe that you can achieve most of your goals by NOT rsyncing the log files. Unless the log files are needed by the system as part of its operation, this would let you;

• have active forwarders continually monitoring the logs on both hot and standby system
• create log files according to your conditional script execution

And you'd only get one copy of the events in the index. As for redundancy if that is an issue, each event will be stored in Splunk and on either system (but not both).

Hope this helps,

Kristian

[kristian_kolb]

glad it helped.

[nvonkorff]

Yep, just exclude the specific log directories from the rsync. Makes total sense. Thanks.