After 6.4 upgrade every server erroring with: ERROR AuditTrailManager Host="host::ServerNAME" cannot open D:\Program Files...\persistentstorage\audit\seqnum_host::ServerNAME.dat for write

[cam343]

Since upgrading the search heads and indexers to v 6.4 (forwarders are still v6.3) the indexers are now logging in splunkd.log the following:

04-07-2016 11:11:15.221 +1000 ERROR AuditTrailManager - Host="host::ServerNAME" cannot open D:\Program Files\Splunk\var\lib\splunk\persistentstorage\audit\seqnum_host::ServerNAME.dat for write, error="The filename, directory name, or volume label syntax is incorrect.".
04-07-2016 11:11:15.221 +1000 ERROR AuditTrailManager - Failed to save seq_no=289 for host="host::ServerNAME" to disk!

This log event is happening for EACH universal forwarder and multiple times. The indexers are Windows2012 and I'm pretty certain Windows ACL's aren't the issue as they have been checked.

On each of the (two) indexers in D:\Program Files\Splunk\var\lib\splunk\persistentstorage\audit there is only one file: seqnum_localhost.dat

What's going on? How can I fix it?

Thanks
Cam - Splunk 6.3 architect

[cps42]

I got this answer from my SE - support didn't get back to me.

I searched our cases and based on responses that support has given to another customer, this is a known issue (SPL-122185) and is targeted to be fixed in 6.4.2.
.
Here is their suggested workaround for another customer - For the time being please attempt following workaround:

1 - create a log-local.cfg (cloned from log.cfg) inside $SPLUNK_HOME/etc, see also here:

http://docs.splunk.com/Documentation/Splunk/6.4.0/Troubleshooting/Enabledebuglogging#log-local.cfg
2 - edit log-local.cfg and add following line inside the [splunkd] stanza (at the bottom will be ok):
category.AuditTrailManager=CRIT
3 - restart splunk.
4 - check splunkd.log again and check if you still the error messages you noticed earlier.

This won't affect any main splunk functionalities. It just will affect the events in the _audit index to not have the sequence ID, while all other info will be there.

Please let me know how it goes, many thanks.

Sorry for the inconvenience.

[cps42]

Added case 369724 as well. Cam, Did you get any further updates on this issue?

This log message doesn't seem to indicate that the indexer is having problems receiving data, as there is new data coming in for hosts, and I am able to execute queries, but the error seems to repeat regularly for each client system that makes a tcp connection to the indexer.

[Richfez]

This sounds like a support ticket.

In the meantime, that filename looks odd. I don't have a Splunk install handy to check against, right now, but seqnum_host::ServerNAME.dat seems like it should be seqnum_localhost.dat. I wonder if ::ServerNAME is supposed to get localhost substituted in? Given that, you could try searching around just to see if some .conf file has ::ServerNAME in it, then maybe compare with an old config from 4.3 and see what it has there...?

[cam343]

Hello,
As mentioned above I do have: "seqnum_localhost.dat" is that folder.
And what the logs are creating is for every forwarder is creating an entry with the server name eg:

04-07-2016 11:11:15.221 +1000 ERROR AuditTrailManager - Host="host::ACME001" cannot open D:\Program Files\Splunk\var\lib\splunk\persistentstorage\audit\seqnum_host::ACME001.dat for write, error="The filename, directory name, or volume label syntax is incorrect.".
04-07-2016 11:11:15.221 +1000 ERROR AuditTrailManager - Failed to save seq_no=289 for host="host::ACME001" to disk!

04-07-2016 11:11:16.221 +1000 ERROR AuditTrailManager - Host="host::ACME002" cannot open D:\Program Files\Splunk\var\lib\splunk\persistentstorage\audit\seqnum_host::ACME002.dat for write, error="The filename, directory name, or volume label syntax is incorrect.".
04-07-2016 11:11:16.221 +1000 ERROR AuditTrailManager - Failed to save seq_no=412 for host="host::ACME002" to disk!

Also I'm not sure what version 4.3 has to do with it, as it was an upgrade from 6.3...

[Richfez]

Sorry, I meant 6.3.