Hi All, Currently facing an issue in parsing the data. We have customized Technology Add-on app called Test-TA-paloalto is installed on the search head cluster members, but is not working properly. It should be changing the sourcetype of paloalto:network:log to the appropriate sourcetype (threat, traffic, system, etc.). Then it should extract fields based on the new sourcetype.
From props.conf:
[paloalto:network:log]
...
TRANSFORMS-sourcetype = pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_endpoint
From transforms.conf
sourcetype routing
[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
FORMAT = sourcetype::paloalto:network:traffic
As you can see if you search for the paloalto:network:log sourcetype, TRAFFIC does land after 3 commas, but the sourcetype is not changing.
11/8/17
10:11:20.000 AM
Nov 8 07:11:20 host01.XXX.COM1,2017/11/08 07:11:19,007257000034869,TRAFFIC,start,0,2017/11/08 07:11:19,10.134.64.7,168.133.28.172,0.0.0.0,0.0.0.0,trust-test,,,splunk,vsys1,trust,test,ethernet1/2,ethernet1/1,Splunk,2017/11/08 07:11:19,195748,1,38754,8089,0,0,0x4000,tcp,allow,1952,382,1570,6,2017/11/08 07:11:20,0,any,0,76095011,0x0,10.0.0.0-10.255.255.255,United States,0,3,3,n/a,0,0,0,0,,host01,from-policy,,,0,,0,,N/A
eventtype = nix-all-logs eventtype = pan network host = host01.XXX.COMindex = firewall linecount = 1 punct = __::..,//::,,,,,//_::,...,...,...,...,-,,,,,,, source = /opt/syslogs/paloalto/host01.XXX.COM/paloalto.log sourcetype = paloalto:network:log splunk_server = splunk01 tag = network
Kindly let me know from where/how to investigate this issue and fix the parsing.
Hi Somesoni2, thanks for your effort on this, We had fixed the issue by editing the props.conf configured with the TZ = PST8PDT and placed in all the HF instance. When we changed the TZ = UTC parsing issue got fixed.
Hi Somesoni2, thanks for your effort on this, We had fixed the issue by editing the props.conf configured with the TZ = PST8PDT and placed in all the HF instance. When we changed the TZ = UTC parsing issue got fixed.
Since you were able to correct the issue, would you mind posting the answer and accepting it so that others can easily find the solution?
The data parsing happens on the indexer or heavy forwarder , whichever comes first in the data flow. Where are you collecting the data? If it's not search head, then your TA apps should be deployed to your Indexer/heavy forwarder.