Two (same) devices sending syslogs indexed with different time by splunk

[gaepea]

Hej,

I have two juniper switches (same hardware model running same OS version) configured to send their syslog to Splunk.
These two units - let's call them A and B - have the same time configured (with ntp).

At first, I thought splunk was receiving the syslog from one but not from the other.
But then broadening my search in "all time" I found out that splunk actually do receive the syslogs from both but:
- for switch A, it is indexed with the correct time and date (let's say 2019 11 07 11h46)
- for switch B, it is indexed with exactly 1 year difference (i.e. 201*8* 11 07 11h46)

I cannot find any reason for that as the two units are the same and configured the same.

Can anyone help me on how I can troubleshoot that ?
Unless there is a trivial reason I did not think about ?

By the way, I have a number of other juniper units (~ 50 units) that are sending their syslog as well previoulsy configured and it seems to work correctly for them - I did not check one by one though. I just noticed this problem with these two units I added recently (yesterday).

Thanks in advance.

Best,
Gae

[gaepea · Accepted solution]

Right so, in case it can help someone in the future, here are my findings.

As said, the syslog message does not contain the year. Then it happened that Splunk actually took the last part of the IP of the device (.18) and used it as the year (=2018) for some reason. I noticed that actually all my devices with IPs ending with .14 .15 .16 .17 and .18 had this problem.

One way to "solve" it is to use props.conf.

Solution 1

[host::(*.14|*.15|*.16|*.17|*.18)]
DATETIME_CONFIG = CURRENT

This is not really good because it replaces the timestamp with the time splunk receives the message if I am correct.

Solution 2

MAX_DAYS_AGO = 363

There is seems it keeps the timestamp of the syslog message and just set the year to 2019.

This is still pretty weird for me and looks more like a bug (???)

Any additional comment is welcome.

[gaepea]

I did a tcpdump to capture the message that arrives. It seems the timestamp in the syslog message sent by A is correct and looks the same as the one sent from B. It complies with the RFC3164. Especially according to RFC3164, the year is not even specified, I quote:

The TIMESTAMP field is the local time and is in the format of "Mmm ddhh:mm:ss" (without the quote marks) where:

     Mmm is the English language abbreviation for the month of the
     year with the first character in uppercase and the other two
     characters in lowercase.  The following are the only acceptable
     values:

     Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec

     dd is the day of the month.  If the day of the month is less
     than 10, then it MUST be represented as a space and then the
     number.  For example, the 7th day of August would be
     represented as "Aug  7", with two spaces between the "g" and
     the "7".

I am really puzzled here. What could splunk do with it that it ends up in 2018 instead of 2019 ?

12:27:42.288906 IP 194.47.252.18.514 > 192.168.19.219.22221: SYSLOG local7.info, length: 121
0x0000:  4500 0095 65cd 0000 3f11 82c5 c22f fc12  E...e...?..../..
0x0010:  c0a8 13db 0202 56cd 0081 b541 3c31 3930  ......V....A<190
0x0020:  3e4e 6f76 2020 3720 3132 3a32 373a 3432  >Nov..7.12:27:42
0x0030:  2077 2d6b 6972 6b30 312d 652d 3220 6d67  .w-kirk01-e-2.mg
0x0040:  645b 3831 3236 375d 3a20 5549 5f43 4d44  d[81267]:.UI_CMD
0x0050:  4c49 4e45 5f52 4541 445f 4c49 4e45 3a20  LINE_READ_LINE:.
0x0060:  5573 6572 2027 6761 6570 6561 272c 2063  User.'gaepea',.c
0x0070:  6f6d 6d61 6e64 2027 7368 6f77 206c 6f67  ommand.'show.log
0x0080:  206d 6573 7361 6765 7320 7c20 6e6f 2d6d  .messages.|.no-m
0x0090:  6f72 6520 27                             ore.'