Dashboards & Visualizations

Windows audit logs get re-indexed everytime events are written to the monitored file

sebwal10
Explorer

Hello!
I have enabled windows auditing on a windows machine and mounted the directory where all logs are written to on a Ubuntu machine where splunk i installed. I am then monitoring the mounted audit file from the splunk instance. The monitored file is in XML-format, the events are single-line and the last line in the XML-file is always </Events> . Every new event is written before the last line so on the second last line.

The problem is that everytime new events are written to the monitored XML-file, Splunk re-indexes the entire file.
When i search for "index=_internal sourcetype=splunkd component=watchedfile" I get the result "INFO WatchedFile - Checksum for seekptr didnt't match, will re-read the entire file=' /mnt/netapp_audit/audit/audit_splunk_audit_last.xml'.
Other than that, the events are parsed correctly in Splunk.

Why is the entire file re-indexed everytime logs are written to the monitored XML-file?
Is it possible to get Splunk to only read events until the second last line?

FrankVl
Ultra Champion

Can you please edit your question and put the content of the last line between ` characters?

I guess the file actually ends with some xml tag, but due to how this splunk answers forum works, anything like <bla> disappears when it is not posted in code tags.

If the last line contains a closing XML tag, which shifts down, that is why splunk fails on that. Not sure if there is any way to fix that. Really weird logging format to be honest, can this not be configured differently? What kind of logs are we actually talking about?

0 Karma

sebwal10
Explorer

Yes it is a closing XML-tag '/Events'. I will look if there is any way to change the logging format. The logs are NetApp security auditing logs.

0 Karma

FrankVl
Ultra Champion

Yeah, so that messes things up. Splunk keeps track of the last line it has read and expects new logs to be added after that. In your case, that last line is the line with </Events>. When a new event is added, it is added on that same line and the </Events> is shifted down. So splunk detects that the last line it had already read is now changed and that triggers re-reading the entire file.
I don't see any settings in inputs.conf that can change that behavior, so I guess looking at the data source to see if it can log in a different way is your best bet.

I've upgraded my comment to an answer, so you can accept it.

0 Karma

FrankVl
Ultra Champion

When a new event is appended, are older events removed from the file as well? Or does the file keep growing?

0 Karma

sebwal10
Explorer

It keeps growing

0 Karma

rasmusbrunzell
Engager

I have the same problem but with NetApp security auditing logs.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...