Input data getting source type changed

[Jon_Irish]

I used to have a PaloAlto firewall and i had it setup to syslog on ump/5514. I was also running a couple of PaloAlto applications. I have retired the PaloAlto firewall and I uninstalled the apps via the "splunk remove app [appname] -auth :" command. I have recently installed a pFsense firewall in its place, and it to is setup to syslog via udp/5514. I am ingesting the new syslog data fine, but all of it is getting tagged with a source type of "pan:log". This is what the old PaloAlto data was tagged with so it worked with the PA applications. I have verified that my Data Inputs setting for udp/5514 is set to use a source type of "pfsense_syslog". Thus, something is overriding this. I have searched my system for a non-default transforms.conf, but all I see are the "default" examples.

Any ideas where I can look to determine what is causing this?

Thanks!
Jon

[mattymo]

Hey Jon_Irish,

Can you check the output of ./splunk btool inputs list udp://5514 --debug? Or just ./splunk btool inputs list udp --debug

[splunker@n00bserver bin]$ ./splunk btool inputs list udp://5514 --debug
/home/splunker/splunk/etc/apps/launcher/local/inputs.conf            [udp://5514]
/home/splunker/splunk/etc/system/default/inputs.conf                 _rcvbuf = 1572864
/home/splunker/splunk/etc/apps/launcher/local/inputs.conf            connection_host = ip
/home/splunker/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf evt_dc_name = 
/home/splunker/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf evt_dns_name = 
/home/splunker/splunk/etc/system/local/inputs.conf                   host = n00bserver
/home/splunker/splunk/etc/system/default/inputs.conf                 index = default
/home/splunker/splunk/etc/apps/launcher/local/inputs.conf            source = syslog
/home/splunker/splunk/etc/apps/launcher/local/inputs.conf            sourcetype = syslog

Or in the GUI (if standalone) Settings > Data Inputs and check for the 5514 config.

That sourcetype is set in the inputs.conf, then the pan:log sourcetype is in the props.conf, which you can look for with ./splunk btool props pan:log --debug`

I assume that because its coming in on the same listener, it is simply applying the same settings??

[Jon_Irish]

Thanks for the suggestions mmodestino,
I tried all three suggestions, but nothing really grabs my attention:

# ./splunk btool props list pan:log --debug ==> no output
# ./splunk btool inputs list udp://5514 --debug ==> no output
# ./splunk btool inputs list udp --debug
/Applications/Splunk/etc/system/default/inputs.conf              [udp]
/Applications/Splunk/etc/system/default/inputs.conf              _rcvbuf = 1572864
/Applications/Splunk/etc/system/default/inputs.conf              connection_host = ip
/Applications/Splunk/etc/system/local/inputs.conf                host = Jons-iMac.local
/Applications/Splunk/etc/system/default/inputs.conf              index = default
/Applications/Splunk/etc/apps/TA-pfsense_a3sec/local/inputs.conf [udp://192.168.1.2:5514]
/Applications/Splunk/etc/system/default/inputs.conf              _rcvbuf = 1572864
/Applications/Splunk/etc/apps/TA-pfsense_a3sec/local/inputs.conf connection_host = ip
/Applications/Splunk/etc/system/local/inputs.conf                host = Jons-iMac.local
/Applications/Splunk/etc/apps/TA-pfsense_a3sec/local/inputs.conf index = gw_pfsense
/Applications/Splunk/etc/apps/TA-pfsense_a3sec/local/inputs.conf sourcetype = pfsense_syslog
/Applications/Splunk/etc/apps/search/local/inputs.conf           [udp://514]
/Applications/Splunk/etc/system/default/inputs.conf              _rcvbuf = 1572864
/Applications/Splunk/etc/apps/search/local/inputs.conf           connection_host = ip
/Applications/Splunk/etc/system/local/inputs.conf                host = Jons-iMac.local
/Applications/Splunk/etc/apps/search/local/inputs.conf           index = main
/Applications/Splunk/etc/apps/search/local/inputs.conf           sourcetype = syslog
# ./splunk btool props pan:log --debug
Invalid command: pan:log

Thanks!
Jon

[mattymo]

looks good, did you restart splunk?

./splunk restart

[jon_d_irish_ctr]

After restarting, it appears that the sourcetype is now correct. Odd that a restart was required. I would have thought that I would have been notified of a need to reboot when I uninstalled the applications. Oh well, all is well now. Thanks for the help!

Jon

[mattymo]

ah nice! now I can sleep better at night! 😉

Good reference here regarding what config changes require restart, cause editing the conf files won't alert you to needing to restart...

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationfilechangesthatrequirerestart

[jon_d_irish_ctr]

LOL 😉