Getting Data In

How to to extract fields from Squid logs to Splunk from PFsense Firewall using 2.2.1?

stilesak
New Member

I have standard UDP logs from PFsense being sent to my Splunk server. However, I can't seem to get the Squid logs to Splunk. Does anyone have some quick advice on how to get these from a PFsense device?

Thanks

Apr 26 21:00:51 192.168.1.1 Apr 26 21:00:51 (squid-1): 1430110851.655 69 192.168.2.51 TCP_MISS/200 638 GET http://geo.yahoo.com/p? - ORIGINAL_DST/208.71.44.31 image/gif
host = 192.168.1.1 source = udp:514 sourcetype = squid

Tags (3)
0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You need to check the source of the file. Seems you are logging the Date and timestamp twice. Typically this is a r/syslog/-ng related issue for the input.

You can check the props for the squid source and change the parsing rules so it ignores the second date and timestamp, or fix the source...

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

It's worse than that Eric... the app is very old and is written were it MUSt have all the fields in order for the transforms to work ( all in a row). On top of the syslog header... the event is also missing three or four fields (and I can't tell what the missing one is because there should be nothing there according to the transforms... ) it also doesn't account for the existence of the timestamp at all and begins with a ^ and expects the duration to be there... even without syslog, that wouldn't work, because the timestamp is going to be there either way... 😕

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

rsennett_splunk
Splunk Employee
Splunk Employee

I've edited your question so that others can be helped here... Initially it seemed as if you weren't getting the data in, but in the end you're describing a problem with extracting the fields.

The fields are there but the app you're using isn't accounting for the syslog header. (the Squid App for Splunk Enterprise seems to have been updated in 2011 and not again...)

Also it's also counting on certain values being present in order to pick up the fields.. Look at the transforms.conf in the app and you'll see that the regex is not going to match your events.
So if you like, you can change the props.conf from:
REPORT-squid = squid
to

EXTRACT-squid =  \(squid-1\):\s+(?P<epoch_time>\d+\.\d+)\s(?P<duration>\d+)\s(?P<clientip>[^\s]+)\s(?P<action>[^\/]+)\/(?P<http_status>\d{3})\s(?P<bytes>\d+)\s+(?P<method>[^\s]+)\s(?P<uri>[^\s]+)\s(?P<unknown_value>[^\s]+)\s(?P<hierarchy_code>[^\/]+)\/(?<server_ip>\d+\.\d+\.\d+\.\d+)\s(?P<content_type>.+)

And you'll be able to get the fields. Or if you feel comfortable, you can alter the transforms, conf and use the regex I've given you (removing or accommodating for the FORMAT directive.

Note the savedsearches.conf in the app... all searches hang on the presence of the action field. You probably didn't have any fields.
action=* means... pull only events where the action field exists... that would be 0 until you extract the field.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

stilesak
New Member

Thanks for the info. I do have have the logs getting to Splunk but the problem seems that Squid for Splunk does not seem to be parsing the data correct for the dashboards.

Search: sourcetype=squid returns the following entries..

Apr 26 21:00:51 192.168.1.1 Apr 26 21:00:51 (squid-1): 1430110851.655 69 192.168.2.51 TCP_MISS/200 638 GET http://geo.yahoo.com/p? - ORIGINAL_DST/208.71.44.31 image/gif
host = 192.168.1.1 source = udp:514 sourcetype = squid

0 Karma
Get Updates on the Splunk Community!

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...