homemonitor app stops indexing after snort app install

[ShaunBaker]

Hello all, after installing the SnortforSplunk app, I deviated from the instructions as I already had Homemonitor installed, so of course could not use the SnortforSplunk instructions of using UDP 514. Using a different port, the SnortforSplunk app is feeding in- refreshing the indexes page on the Splunk webUI shows data is flowing in/indexing for the 'snort' index, but the homemonitor index sh*ts the bed. Every time I reboot the indexer, the homemonitor index works briefly and gets some data, but then stops. I have homemonitor recieving from UDP 514 and had been working for months, with snort I tried 992, then deleted that data input and went with TCP 49152-- both coming from the same pfsense host IP address.

Can PFsense not support sending two different logs (syslog to UDP 514, barnyard2 over XXX) from the same host IP?

I can't see an easy way to just send both over UDP 514 and using transforms for the sourcetype organization.

Maybe I could modify the XML and import the snort app functionality over into homemonitor?

[ShaunBaker]

I changed the pfsense settings to send all system and snort logs over syslog 514, nuked my splunk install from orbit, installed 7.x.x, re-installed the home monitor app. Thinking of just bringing over some of the field extractions from the snort app, and some of the snort app panels with corrected search strings for how home monitor labels the fields (source_ip vs. src_ip etc).

[ShaunBaker]

Wait, amiracle, your Kam?

[amiracle]

To quote a fellow Splunker, “Kam I am.”

[ShaunBaker]

Awesome! The Homemonitor app is amazing. I think its boiling down to how pfSense is sending out logs- I'm in the pfSense GUI and Splunk webUI and I feel enabling snort, logging, barynard2 etc has changed what comes out of pfSense and why I'm seeing a lot of Homemonitor field extraction not happening anymore- those raw logs are not coming in anymore, its just a bunch of Snort stuff on both UDP 514 and UDP 92152 (guessing, forgot what port I assigned for Barnyard2 log output in pfSense). Is Snort "in front" of the firewalls? I wondering if the very nature of enabling aggressive Snort settings is also diverting a lot of actions that otherwise would have been generating logs your props.conf interacts with...
-- Basically, pfSense is not sending logs out as I had imagined- it can't seem to segregate snort logs to Port X, and pfSense native logs to UDP 514 as specified. Now I don't seem to be getting in any of the stuff that used to get field extracted for your searches, and I'm getting heaps of Snort 'blocked' logs via 514, and still more snort data over port "x" that I specified for the syslog output of barnyard 2. When I go into field extraction myself I can't find a sample of data that your props applies to-- even though in pfsense I still have all the types of logs to be sent checked off.

I cranked down snort to generate less noise and will keep looking to see if "regular" pfsense logs are going over the wire.

[ShaunBaker]

Strange- I changed HomeMonitor's data input from "from list" and "pfsense" to "Manual" and "syslog" and now it seems to be indexing. Interesting that it used to index before as sourcetype 'pfsense' but then after this other app install I have to change that to syslog. Sadly now the HomeMonitor props.conf doesn't seem to be extracting fields anymore...

[amiracle]

The app does look to 514 for syslog data then does a transform based on the source IP or hostname where it converts the source type from 'syslog' to 'pfsense.' Now, if you add another system collecting from 514, then the app will no longer be collecting the data; moreover if it's the same system sending the data (pfsense firewall with snort) then it might complicate the issue.

The best approach would be to setup a syslog box that is collecting the data and split the data sources into folders that the Splunk universal forwarder on the syslog box is monitoring. You can create filters with rsyslog or syslog-ng based on hostnames, ip addresses etc. and have them send to folders. For example:

rsyslog docs
Edit the /etc/rsyslog.conf file and add a filter to send data being sent by pfsense firewall :

if $fromhost-ip startswith '192.168.1.' then /var/log/fios.log
if $fromhost-ip=='0.1.1.1' then /var/log/pfsense.log

You can use a raspberry pi as a syslog server too and run the UF on it: https://www.splunk.com/blog/2013/10/11/introducing-the-splunk-universal-forwarder-for-raspberry-pi.h...

[ShaunBaker]

I am not feeding both syslogs to the same port. PFsense syslog is going to Splunk server IP + UDP 514. Snort is going to Splunk server IP + UDP 92154 (or something like that, I can't recall the port right now).

Is Splunk having an issue collecting two syslogs (though source typed per both app's instruction) from the same IP- but key here, not the same port? I really thought this would be a great way to isolate/properly source type the two streams of data.

I almost feel like sending everything over 514, copying snort for splunk's props.conf data over to home monitors, copy the searches over, and basically build into home monitor another tab for snort for splunk's dashboard.

[ShaunBaker]

I'm still google'ing around- lots of info on giving two different IPs sending to the same port different source types, but nothing on my issue so far. Is there an issue with the same host sending two different logs to separate ports? Should I try un-installing and re-installing home monitor and/or the data input for it? I've looked at both app's inputs.conf and various other .confs, I can't seem to find what the conflict is. I looked at home monitors props and transforms in hopes of finding something that is maybe getting hijacked. The snort for splunk doesn't even have a transforms nor a .conf processing fields before indexing. I don't think this is a PFSense issue because upon every reboot of splunk the home monitor index ingests very briefly, and rebooting PFSense does nothing.