Splunk Search

TImestamps problem

kraveruk
Explorer

Hi Im running the newest splunk, with syslog-ng fifo pipe as a source and logs are coming from around the globe, splunk is in the US so when logs from China are hitting splunk they are like 10h ahead, and they dont show up in search till splunk riches that hour itself

Jun 8 23:37:39 tok-* SYST: Port 29 link active 100Mbs FULL duplex

Jun 8 20:07:40 10.115.1.2 SNTP: The SNTP server parameter value (pool.ntp.org) can not be resolved.

Jun 8 10:37:47 del-## Jun: 8 20:05:42 netTool.sntp: : Failed to sntp request to server 10.**

as you can see logs are coming with local times, and they get indexed like that, now time on the splunk machine is 10:37 and last log shows 2 time zones, i dont have source in props.conf b/c i dont use files to import the logs all i have is syslog pipe and splunk set up to

[fifo:///var/syslog-ng/syslog_fifo]

disabled = false

host = MYHoST

sourcetype = syslog

how can I change that so all of the logs would be logged with 2 timezones, or just logged with the splunk local time instead of sender local time ? thanks

Tags (1)

fox
Path Finder

Did this work? it sounds like you have a similar issue to ours. One way to stop the timestamp from being auto adjusted by the search head is to change the TZ at index time in the props file to that of the search head location. This does have side effects though as the local data will be theoretically indexed incorrectly.

0 Karma

fox
Path Finder

It seems for now that there is no way of disconnecting the TZ from a timestamp. I have effectively designed the architecture to resolve this issue. There will be a different instant of the application per region and timestamps will be indexed with the correct TZ to allow future scalability...

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

If you just want to use arrival time for this source rather than extracted time you can set in props.conf:

[source::/var/syslog-ng/syslog_fifo]
DATETIME_CONFIG = CURRENT

If you can detangle the different timezones into different files, you can set in props.conf:

[source::/var/syslog-ng/host1/syslog_fifo]
TZ = <host1 timezone>

[source::/var/syslog-ng/host2/syslog_fifo]
TZ = <host2 timezone>
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...