I'm trying to modify the below Splunk app to perform additional sourcetype extraction. TA-Pfsense App
I have data coming in over syslog, and being saved as sourcetype "pfsense."The TA performs a transforms-extract on the pfsense sourcetype in props.conf based on regex in transforms.conf that looks for a timestamp at the beginning of the event. It then extracts the pfsense log type(ex. filterlog, dhcpd, openvpn), which is typically after the timestamp, and sets it as the sourcetype.
transforms.conf
[pfsense_sourcetyper]
REGEX = \w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s(?:[\w.]+\s)?(\w+)DEST_KEY = MetaData:SourcetypeFORMAT = sourcetype::pfsense:$1
props.conf
[pfsense]
TRANSFORMS-pfsense_sourcetyper = pfsense_sourcetyper
SHOULD_LINEMERGE = falseSEDCMD-event_cleaner = s/^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+\S+\.\S+\s+/\1/g
SEDCMDevent_cleaner2 = s/^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s)+/\1/g
SEDCMD-event_cleaner3 = s/^\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s\S+\s(\S+\s)/\1/g
So the above from the TA works fine, except I configured a snort feed in Unified2 format that I want to be able to send to pfsense:snort sourcetype, but it doesn't work because it has a completely different format. I tried to tweak the TA by adding in another transforms-extract but it does not work. I've tried different variations of regex to match the log format but I've been unable to get it to work so far. Any thoughts?
Raw Log(IP's redacted): | [SNORTIDS[LOG]: [pf.local] ] || 2020-05-27 16:15:31.157+000 2 [1:2403468:57488] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 85 || misc-attack || 6 89.XXX.XXX.XXX 2XX.XXX.XXX.XXX 4 20 0 40 1773 0 0 17229 0 || 51267 4303 1181682881 0 5 0 2 1024 1546 0 || 64 ..g.KlL..p....E..(......CMY...d....C..Fo......P................. ||
Added to transforms.conf
[pfsense_snort]
REGEX = (?:\| \[)(SNORT)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::pfsense:snort
Added to props.conf
[pfense:snort]
TRANSFORMS-pfsense_snort = pfsense_snort
... View more