Monitoring a file that is continiouisly written to

[horsefez]

Hi fellow splunkers,

I got the task to monitor a file on a system that gets created on serverstart and then gets written to for the time the server runs (6 months or more)

How am I able to monitor such a large file if new lines get added?
I heard about "follow tail", but anyone seems to discourage you to do that.

Any idea how this could be possibly done?

Thanks in advance!
Best regards,
pyro_wood

[woodcock]

What is wrong with a simple monitor stanza? This is exactly what Splunk is designed to do. How is your situation in any way complicated/non-standard?

[horsefez]

Looks like you are correct. Had a faulty monitor-stanza!
Thanks!

[skoelpin · Accepted solution]

If I'm understanding the question correctly, you want to monitor a file continuously and send that data to Splunk? If so then you can install a universal forwarder to monitor that file. Each time that file has new data, the Splunk forwarder will see this and forward data to the indexer.

So an example would be, you have a file that sits on a file system and gets written to when it starts and if there is data requested from the server. Once this file is written to, the Splunk forwarder will see this and pick up all new changes and forward it to Splunk while disregarding everything it's already indexed from that file. You can write to the file as little or frequently as you want to

If you want to monitor a file, your inputs.conf will be located in \etc\system\local and look like this

[monitor//C:\PATH_TO_FILE]
disabled=false
sourcetype=YOUR_SOURCETYPE
index=YOUR_INDEX

And your output.conf will point to the indexer and will look like this

[tcpout]
defaultGroup = INDEXER_IP_9997

[tcpout:INDEXER_IP_9997]
server = INDEXER_IP:9997

[tcpout-server://INDEXER_IP:9997]

[horsefez]

Thank you skoelpin,
I did make an error in the monitor-stanza so the file didn't get read correctly. So I wondered if splunk wasn't able to do this task. Thanks to your example I was able to realize and fix this!

[gcusello]

What is your problem? Can you explain it?
Splunk usually manage upgrade of a single file with insert of new lines, without particular configurations.
Are you using a Forwarder or is a local file?

Bye.
Giuseppe

[horsefez]

Hey cusello,
you are right. It was my fault, had a faulty monitor-stanza!