Network downtime breaks A3Sec app ingestion

[ShaunBaker]

I have a different (and mislabeled) post about this that is labeled answered, but the issue is not.

I don't know if I found a bug- or more likely I'm really bad at hunting down a simple issue, but I found if my router/switch goes down (APC took a dump) for a few hours, upon restoring the network the A3Sec app (for pfsense logs) will start to get the following type of errors in the splunkd.log:

06-27-2018 19:23:58.543 -0700 WARN DateParserVerbose - A possible timestamp match (Sat Setp 8 18:46:43 2001) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Contex: source=udp:514 | host xxx.xxx.x.x | pfsense_syslog |


06-26-2018 20:51:26.834 -0700 WARN DateParserVerbose - Failed to parse timestamp in the first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to time stamp of previous event (Tue June 26 08:27:00 2018). Context: source=udp:514 | host =xxx.xx.x.x | pfsense_syslog

This is on a CentOS7 VM, ESXi 6.4. The CentOS7 box gave itself an IPv6 address when the network went down, had to get it back to it's IPv4 when the network got back up.

I reviewed the app's props.conf, transforms, multiple reboots, deleted the gw_pfsense index, made it again, clean indexes (I guess this is clearing the fishbucket on 7.x?), turn pfsense syslog output on and off, reboots etc to no avail.

Ultimately I went back to an earlier snapshot and now gw_pfsense is indexing firewall events again.

[ShaunBaker]

Interesting observation, the snaptshot did not have SnortforSplunk setup. When I set that back up, but with the data input UDP 1514 (both barnyard in pfsense and input in Splunk), it breaks AS3sec (SnortforSplunk ingests and extracts). I played with app permission (app, global), re-doing data inputs, sanity checked the inputs.confs of both apps- can't find the issue. I've run into this before, but do not recall what I did to get those two apps / two different UDP inputs to play nice.

With the above logs, SnortforSplunk works, but A3Sec's index stops showing events coming in, and it almost looks like for some reason A3Sec's UDP 514 input starts to use Snort's .confs (hence the time stamp extraction fails).

I'm thinking because SnortforSplunk uses an already built in sourcetype (snort) that is default 514, that a .conf in system, maybe a default or local, is messing up A3Sec's "claim" on 514, but I haven't been able to find this offending .conf.