Why are there Palo Alto App Empty Dashboards on Splunk Cloud?

[prescilianoneto]

Hello,

Our firewall events are flowing to our Splunk Cloud environment however all the events have the sourcetype pan:log instead of pan:traffic, pan:config, pan:threats, etc. This results in empty Palo Alto App dashboards.

I tried to ask for Splunk Cloud support, however they told me that "the Palo Alto app and add-on are not splunk supported".

My guess is that the Palo Alto Add-on is not installed on the indexers. The "Manage Apps" Splunk menu shows only the App, not the Add-on. I know that the add-on is installed because it appears in the main menu. I would try to uninstall and reinstall the add-on by myself but I don't have access to.

Most of the Palo Alto documentation refers to a single instance environment, so I'm not sure about how to do solve this issue in Splunk Cloud.

Any advice? It seems that I need to ask the Splunk Cloud Support guys exactly what they need to do to solve the problem, and I'm neither a Splunk nor Palo Alto expert.

Best Regards,

Presciliano

[btorresgil]

If you're using a universal forwarder with syslog-ng, the default configuration can cause this problem. Make sure you configure syslog-ng not to add headers to the syslogs. More information and configuration examples here:

https://splunk.paloaltonetworks.com/universal-forwarder.html

[micahkemp · Accepted solution]

You need to have a Splunk Heavy Forwarder that you run (not in Splunk Cloud) run the Palo Alto TA, have it be the ingest point for the Palo Alto logs, then have it forward those logs to the Splunk Cloud indexers.

This way your index-time transforms can run (just before they get to the Splunk Cloud indexers).

[prescilianoneto]

But I already have a Universal Forwarder, is really need to upgrade it? I forgot to mention that dashboards worked fine after Palo Alto App and Add-on initial installation, they stopped working after some time.

[micahkemp]

Universal Forwarders won't do the transforms the Palo Alto app requires.

[prescilianoneto]

Hello micahkmep, I upgraded my Universal Forwarder to a Heavy Forwarder and installed both Palo Alto App and Add-on on it. Then, I configured /opt/splunk/etc/apps/Splunk_TA_paloalto/local/inputs.conf as follows:

[udp://192.168.2.100:514]
sourcetype=pan:log
no_appending_timestamp = true

[udp://192.168.2.150:514]
sourcetype=pan:log
no_appending_timestamp = true

I verified that /opt/splunk/etc/apps/Splunk_TA_paloalto/default/props.conf and /opt/splunk/etc/apps/Splunk_TA_paloalto/default/transforms.conf are in place.

So I restarted splunk, but the events are still being presented as pan:log in the search header.

How do I troubleshoot it?

[btorresgil]

Basically what's happening is the parser is not seeing these logs as Palo Alto Networks logs or the parser is never run against the logs. The parser only looks at the first 4 fields of the log to make this determination. Are you sure you're using the default syslog format on the firewall/panorama? Can you offer a screenshot of the logs you see coming in as sourcetype pan:log?

[prescilianoneto]

Hello, I realized that my forwarder was acting as a LWF instead of a HF after the upgrade. I reinstalled it from scratch, re-applied the same configuration and then everything started to work fine.

[micahkemp]

Thanks for the update!

[prescilianoneto]

Ok, so I just don't understand how it worked before. I always had a Universal Forwarder.
However, as your answer makes sense, I'll try to upgrade it.
Thank you