Getting Data In

Events have wrong timestamp. How to correct time configuration?

vetash
New Member

Hi all! Sorry, if this question was already asked by someone, but i'm stuck with a time configuration.
So, i just installed Splunk and configured it to listen on UDP port in my network. All hosts send data to it and everything is great, but Splunk shows the wrong time in search results.
This is how i see it:
https://www.dropbox.com/s/e4rf3kxete9qgpv/splunk_f.PNG
Also Splunk shows me the wrong time on all another hosts. Every time I type another ip - Splunk muss time.

This my date on server:
root@monsrv:~# date
Птн Авг 15 09:55:11 IRKT 2014

What do I need to configure to see the right time in search results?

Sorry for my bad English. Hope you understand me. 🙂

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

You should read this http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/Aboutconfigurationfiles and the following couple of pages.

0 Karma

strive
Influencer

This is direct from splunk documentation:

By default, Splunk Enterprise applies time zones using these rules, in this order:

  1. Splunk Enterprise uses any time zone specified in raw event data (for example, PST, -0800).

  2. Splunk Enterprise uses the value of a TZ attribute set in props.conf, if the event matches the host, source, or source type specified by the stanza.

  3. If an event that arrives at an indexer originated at a forwarder, and both the forwarder and the receiving indexer run Splunk Enterprise 6.0 or later, then Splunk Enterprise uses the time zone that the forwarder provides.

  4. Otherwise, Splunk Enterprise uses the time zone of the server that indexes the event.

Note: If you change the time zone setting in the system Splunk Enterprise runs on, you must restart Splunk Enterprise for it to pick up the change.

So in your case:

Point 1 is not applicable as your events do not contain time zone information.

Point 2 is also not applicable. Since you have not modified any props.conf settings. Also, you are not aware which props.conf contains the settings.

Then in that case either point 3 or point 4 is applicable. Since you have mentioned in your comment that both the host and the system(receiver) both are in UTC+9 timezone. That timezone is considered for indexing events.

What you need to do is this:

Step 1: Create props.conf file under /opt/splunk/etc/system/local/ directory of your indexer. The full path will look like this /opt/splunk/etc/system/local/props.conf on indexer node.
Note: You can also create this props.conf file under /opt/splunk/etc/apps/<your_app>/local/ directory. Here your_app is the dedicated app that you have created for your indexer node.

Step 2: Add a stanza with your sourcetype

[Your_Sourcetype]

Note: you can have stanza with source, host and sourcetype. I have chosen sourcetype here.

Step 3: Under that stanza specify the timezone

[Your_Sourcetype]
TZ = UTC

For more information on setting timezones read http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Applytimezoneoffsetstotimestamps

0 Karma

vinodmadaan
Path Finder

HI Strive,

Thanks for your response, I am facing the same problem but with a weird twist that is this problem is not for all the records, I am having few records (about 100) that are having the this time stamp issue and rest are absolutely fine. Any idea what could possibly be the reason for that?

Thanks,
Vinod.

0 Karma

vetash
New Member

So where is required props.conf?
root@monsrv:~# find /opt/splunk/ -name props.conf

/opt/splunk/etc/apps/search/default/props.conf
/opt/splunk/etc/apps/legacy/default/props.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/props.conf
/opt/splunk/etc/apps/learned/local/props.conf
/opt/splunk/etc/apps/sample_app/default/props.conf
/opt/splunk/etc/system/default/props.conf

0 Karma

vetash
New Member

Yes UTC+9. Timezone on system is right, also on the hosts who sending logs for splunk. On sceenshot on right side is actual date. Splunk shows incorrect date. (mark as circle 🙂 ) And i have no idea where i need to config it.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Looking at the timestamps in your screenshot it seems this is a timezone issue. What time zone is the source and your user in? Your server seems to be in UTC+9?

Also, who's prepending the timestamp and host to the syslog event? Is your Splunk doing that, or is that already prepended before it gets to Splunk? If that's prepended before it gets to Splunk, what timezone is that system in?

0 Karma

strive
Influencer

Haven't you set your configurations in props.conf file. Your custom configurations should be under /etc/system/local. If you have written a separate app for heavy forwarder or indexer then the props.conf file should be under that app's local directory.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The file in etc/system/default is useless to us because it only contains default values.

vetash
New Member

Thanks for the reply!
/opt/splunk/etc/system/default/props.conf:
http://pastebin.com/pDzwZA6G

0 Karma

strive
Influencer

Post your props.conf configurations.
What values you have set for TIME_FORMAT, TIME_PREFIX

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...