Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

[jplumsdaine22]

We are trying to index NetApp XML audit logs. The look like this

<Events xmlns=blah>
<Event>stuff</Event>
<Event>stuff</Event>
<Event>stuff</Event>
<Event>stuff</Event>
</Events>

Unfortunately, new events are INSERTED BEFORE the final tag.
So the next time it reads the file, the value of scrc will be different and Splunk reindexes the entire file. The error message is "Checksum for seekptr didn't match, will re-read entire file"

This is expected behaviour from Splunk, but I'm wondering if anyone has managed to work around it? One method that comes to mind is can the seekptr be told to ignore a regex? If it ignored the final element, then there would be no scrc mismatch.

Any ideas?

[kapanig]

Did you ever fix this? I have tried (to no avail):

INPUTS (only monitoring last .xml file)

[ontap]
initCrcLength = 2048
multiline_event_extra_waittime = true
disabled = 0
sourcetype = ontap
index = ontap

PROPS:

[ontap]
SHOULD_LINEMERGE = false
KV_MODE = xml
LINE_BREAKER = ()
MUST_BREAK_AFTER = \
TRANSFORMS-t1 = remove_header_footer

TRANSFORMS:

[remove_header_footer]
REGEX=^<(\/|)Events(\s|>)
DEST_KEY = queue
FORMAT = nullQueue

Still getting

WatchedFile - Checksum for seekptr didn't match, will re-read entire file=(.xml file)
WatchedFile - Will begin reading at offset=0 for file= (.xml file)

[kapanig]

CORRECTION:
LINE_BREAKER = (<Event>)
MUST_BREAK_AFTER = \</Event\>

[jplumsdaine22]

We ended up reading the rotated log files instead of the live file, as there is no way to manipulate seekptr

[geraldomagella · Accepted solution]

Hello! Have you fixed it?

Are you monitoring the files directly? How about exclude the "filerXYZ-last.xml" and getting only the one that was already rotated? That could fix your issue, right?
You could set it to rotate every 5 minutes (or something like that) and keep only the 10, 15 files (loglimit).
If you need the cDot commands, I can help you with this.

[jplumsdaine22]

Hi,

We did in fact end up reading the rotated file. Works fine, but we miss being able to get real time info.