Getting Data In

How do I add the right time stamp and correct server name to the following syslog messages?

saadi381
New Member

Hi guys,

Please pardon my ignorance here as i am new to Splunk. I am using Splunk 7.1 on a Windows server and forwarding syslog messages from Linux server (j01ftc) , now as you can see below example

<0>Oct 18 14:49:51 j01ftc Oct 19 02:02:23 akl-ftc-sbc3b.vfnz-ipsn akl-ftc-sbc3b sipd05[4f] ERROR could not identify psipcontact

There are 2 issues. One is that the wrong time stamp is in beginning of the message and 2nd server name added , i want to exclude both of these from my logs in Splunk or at least fix the time in the event.

0 Karma

FrankVl
Ultra Champion

I would very much suggest fixing this on syslog side, instead of pulling all kinds of tricks on Splunk side to fix problems introduced by the syslog setup.

What does your syslog setup look like? Because the duplicate headers seem to indicate there is some intermediate syslog server involved or so? And also: how exactly are you feeding this into Splunk?

0 Karma

saadi381
New Member

The feed is like
device-->(tcp514)syslog server -->TCP9998 splunk.

i highly doubt the duplicate time stamp is added by Syslog server , i believe its added on splunk server

0 Karma

FrankVl
Ultra Champion

Unless you specifically configured Splunk to add that header, I don't think so. Splunk only does that by default for UDP inputs and even then, it does not add that <xx> part, only timestamp and sending host.

You can validate that by running a networkcapture (e.g. using tcpdump) on either the outgoing traffic of the syslog server, or the incoming traffic on the splunk server. If the header is there already, it isn't Splunk who is adding it.

Feel free to share your relevant Splunk configs (inputs, possible props/transforms) and syslog daemon configs here for further troubleshooting.

0 Karma

ehollima
Path Finder

sudosplunk's answer is the solution that worked for me using your example string:
(Oct 18 23:38:53 j01ftc Oct 19 12:38:46 akl-ftc-sbc3b.ipsn akl-ftc-sbc3b sipd01[4b] ERROR could not identify psipcontact)

[sourcetype]
TIME_PREFIX = j01ftc\s
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15

0 Karma

sudosplunk
Motivator

Hi @saadi381,

That is the syslog header added by syslog server for every event.

If you want to extract the second timestamp (Oct 19 02:02:23), configure the props.conf as below:

[sourcetype]
TIME_PREFIX = j01ftc\s
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15

If you want to completely remove the syslog header from events then, configure the props.conf as below:

    [sourcetype]
    SEDCMD-dropSyslogHeader = s/(^[\w\s\:]+j01ftc\s)//g
    TIME_PREFIX = ^
    TIME_FORMAT = %b %d %H:%M:%S
    MAX_TIMESTAMP_LOOKAHEAD = 15

I would suggest using 1st props.conf as it is much simpler. HTH!

0 Karma

saadi381
New Member

under D:\Splunk\etc\apps\search\local props.conf

[saad_test]
NO_BINARY_CHECK = true
TZ = Antarctica/South_Pole
category = Custom
pulldown_type = 1
disabled = false
SHOULD_LINEMERGE = false
SEDCMD-dropSyslogHeader = s/(^[\w\s:]+j01ftc\s)//g
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 15
INDEXED_EXTRACTIONS =
DATETIME_CONFIG =

but still no luck 😞

looking like

<0>Oct 18 23:38:53 j01ftc Oct 19 12:38:46 akl-ftc-sbc3b.ipsn akl-ftc-sbc3b sipd01[4b] ERROR could not identify psipcontact

0 Karma

sudosplunk
Motivator

Few questions please,
Can you tell me if <0> is part of the event or is it something that got added while posting the question?
Can you tell more about your architecture?

UF (on syslog server) --> Indexer then, these configs should be on indexer.
HF (on syslog server) --> Indexer then, these configs should be on HF.

Make sure that these configuration are present on parsing layer (HF and Indexer).

Finally, did you try the 1st props.conf. It will fix the timestamp but won't discard syslog header.

Also, as @FrankVl suggested, you should have a look at syslog configuration to troubleshoot further.

0 Karma

iamarkaprabha
Contributor

You can overwrite those using props.conf and transform.conf

0 Karma

saadi381
New Member

would you be able to point me to ,how to remove "Oct 18 14:49:51 j01ftc " from my log messages

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...