All Apps and Add-ons

Cisco syslog and double timestamp

templier
Communicator

Hello, all.
I have a new question.

That we have:
1. Main splunk server
2. Installed Cisco Security Suite and Splunk Add-on for Cisco ASA
3. Configured inputs data from cisco on UDP (create this via browser). Set index and sourcetype cisco:asa
4. Two cisco asa for data semple

And after, when i collected some data, i found one trouble.
For example 2 string:

From first cisco:

Sep 11 17:25:45 xxx.xxx.xxx.xxx Sep 11 2017 17:25:46: %ASA-3-713902: Group =
yyy.yyy.yyy.yyy, IP = yyy.yyy.yyy.yyy, Removing peer from correlator table
failed, no match!

And from second:

  Sep 11 17:27:00 yyy.yyy.yyy.yyy %ASA-3-710003: TCP access denied by ACL
    from xxx.xxx.xxx.xxx/54483 to INT-WAN2:xxx.xxx.xxx.xxx/22

And how you can see on first cisco i have double timestamp, but on second cisco all good.

I dump traffic to splunk and all cisco send correct identical data to my udp.

How i can fix it?
Thanks!

0 Karma
1 Solution

jtacy
Builder

It sounds like the logging timestamp option is enabled on the ASA. Consider running no logging timestamp on the ASA to disable this behavior. From the docs:

logging timestamp
To specify that syslog messages should include the date and time that the messages was generated, use the logging timestamp command in global configuration mode. To remove the date and time from syslog messages, use the no form of this command.
Source: https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/l2.html

View solution in original post

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Hey @templier, If @jtacy solved your problem, please don't forget to accept an answer! You can upvote posts as well. (Karma points will be awarded for either action.) Happy Splunking!

0 Karma

templier
Communicator

Yeah, of course, after solved)

0 Karma

jtacy
Builder

It sounds like the logging timestamp option is enabled on the ASA. Consider running no logging timestamp on the ASA to disable this behavior. From the docs:

logging timestamp
To specify that syslog messages should include the date and time that the messages was generated, use the logging timestamp command in global configuration mode. To remove the date and time from syslog messages, use the no form of this command.
Source: https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/l2.html

0 Karma

templier
Communicator

Hello
I haven't access to cisco for test for check it now.
But i want clarify one moment, timestamp in alert - it's timestamp without splunk timestamp (RAW data).
And how i understand - command logging timestamp disable timestamp field on all syslog from cisco?

0 Karma

jtacy
Builder

Sorry for the delay! I had to do some testing with the Splunk UDP server since I normally receive syslog using syslog-ng. It appears that while syslog-ng attempts to use the ASA-provided timestamp (rewriting the message to place the timestamp at the start of the line), Splunk seems to prefix the message with the current timestamp. The cisco:asa sourcetype looks for the timestamp at the beginning of the message so Splunk ends up using the current timestamp.

One possible improvement here is to use the no_appending_timestamp option on the UDP input. It looks like you may have to set this in inputs.conf manually. This will stop Splunk from adding a timestamp and host to the beginning of each syslog message. Splunk will end up using the timestamp provided by the ASA which is great. On the other hand, it sounds like you might have one ASA that isn't sending timestamps so that one would need to be configured to send timestamps.

It looks like setting no logging timestamp on the ASA is also still a possible option. However, I would note that the field extractions seem to work fine even with the additional timestamp. The biggest problem is that Splunk is writing the wrong timestamp to the index. You could potentially update the timestamp extraction for the cisco:asa sourcetype in props.conf to fix that.

Now that I've considered this further, if I wasn't able to use syslog-ng I would personally set no_appending_timestamp on the input and configure each ASA to send both timestamps and hostnames in their messages. I would then configure Splunk to extract the hostname from the syslog message (if it's not doing so already). That seems like the ideal scenario but unfortunately it does require changes on both the sender and receiver.

templier
Communicator

Many thanks for your answer.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...