Getting Data In

CheckPoint FW Logs from Manager with Two Timestamps

wbfoxii
Communicator

We are splunking logs from our CheckPoint FW. The logs are delivered from the CheckPoint manager stations, not directly from the firewalls, so there are two timestamps - first from the management station and next from the FW. We want to index based on the FW time. Here's a typical record:

Jan 17 16:47:31 aaa.bbb.178.200 fw1log: 17Jan2013 17:39:49 accept aaa.bbb.161.11 >eth2c13 rule: nnn; rule_ uid: {93622F88-3071-4FE4-BC7B-9B232AE482E6}; src: aaa.bbb.112.163; dst: aaa.bbb.4.35; proto: udp; product: VPN-1 & FireWall-1; service: 389; s_port: 3490;

The time we want is 17Jan2013 17:39:49.

I went into /etc/system/local/props.conf and added:

TIME_PREFIX = fw1log:\s

TIME_FORMAT = %d%b%Y %H:%M:%S

No luck getting this to use the second (correct) timestamp.

Tags (1)
0 Karma

wbfoxii
Communicator

I changed the time prefix setting to:

TIME_PREFIX = fw1log:\s


(s has a backslash in front of it)

I think there was a global setting from some other application for max timestamp lookahead that needed to be changed from the [default] stanza to a more specific stanza

Anyway, now it is working.

Many thanks to sbrant for keeping me on the right track.

0 Karma

wbfoxii
Communicator

Props.conf is on the indexer, which is also the search head.

This is probably more than you want, but here's btool props list --debug


search [checkpt_log]
system ANNOTATE_PUNCT = True
system BREAK_ONLY_BEFORE =
system BREAK_ONLY_BEFORE_DATE = True
system CHARSET = UTF-8
system DATETIME_CONFIG = /etc/datetime.xml
system HEADER_MODE =
system LEARN_SOURCETYPE = true
system LINE_BREAKER_LOOKBEHIND = 100
system MAX_DAYS_AGO = 2000
system MAX_DAYS_HENCE = 2
system MAX_DIFF_SECS_AGO = 3600
system MAX_DIFF_SECS_HENCE = 604800
system MAX_EVENTS = 256
system MAX_TIMESTAMP_LOOKAHEAD = 128
system MUST_BREAK_AFTER =
system MUST_NOT_BREAK_AFTER =
system MUST_NOT_BREAK_BEFORE =
system SEGMENTATION = indexing
system SEGMENTATION-all = full
system SEGMENTATION-inner = inner
system SEGMENTATION-outer = outer
system SEGMENTATION-raw = none
system SEGMENTATION-standard = standard
system SHOULD_LINEMERGE = True
system TIME_FORMAT = %d%b%Y %H:%M:%S
system TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
system TRANSFORMS =
Splunk_Cis TRANSFORMS-force-sourcetype_for_cisco_devices = force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_wap, fo
rce_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall
system TRUNCATE = 10000
system maxDist = 100

0 Karma

wbfoxii
Communicator

And even in this comment the backslash was removed. I guess I don't know how to markup with Markdown

0 Karma

wbfoxii
Communicator

TIME_PREFIX = ^(?:(?:[^\s]+)\s){5} is what is actually there. I don't know why the formatter removed the backslash in front of the s.

0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

I made a few assumptions about the props.conf. Just to be sure, is the props.conf on the indexer? Does the stanza for the checkpoint data have the correct name to match the sourcetype? Can you post the entire stanza for checkpoint? It's always best to be explicit. Maybe something like this:

[checkpoint]
TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
TIME_FORMAT = %d%b%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 25
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
0 Karma

wbfoxii
Communicator

Couldn't put the answers to your questions in a comment.

0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

Try this as your time prefix:

TIME_PREFIX = ^(?:(?:[^\s]+)\s){5}
0 Karma

Ayn
Legend

extract reload=t was previously a way of reloading SEARCH-TIME properties from props.conf/transforms.conf. Nowadays you don't need that because each search is run in its own process which will read the current props/transforms settings when it starts.

Index-time settings can however NOT be reloaded without restarting Splunk.

0 Karma

wbfoxii
Communicator

That didn't solve it. I assume I can run a query with | extract reload=t at the end to get the new copy of props.conf active. Checked the props list with btool and verified that this is the TIME_PREFIX being used.

Time still matches the first one found.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...