indexing multiple timezone data

[tmuthuk]

We currently have 4 servers that send data to the Splunk indexer. Each server is located in different time zone, Our indexer is in CST timezone. We want to index the data in CST time. Is there anyway it can be before indexing the data.

[martin_mueller]

The best thing to do is to add timezone information to the events at the source. Instead of 2015-10-31 14:15:16.123, make it produce 2015-10-31 14:15:16.123 -0500. Then Splunk will automagically do the right thing.

[woodcock]

"Best" is certainly debatable in the broadest sense. Doing this adds several bytes to each message and costs license and disk space. Granted, it is the most foolproof way to ensure correct timestamping.

[martin_mueller]

I'd take reliable timestamps over a few bytes of license any day of the week.

[woodcock]

Your question is not specific enough to give you a specific answer but let me cover all the bases.

Q1:
How do I modify the raw data so that the timestamp is converted to CST?
A1:
Use this solution while discriminating with host-based stanza headers in props.conf (each host is in a particular timezone so you know how to modify the timestamp for particular TZs).
http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Anonymizedatausingconfigurationfiles
Be sure to also see Q2/A2 and specify TZ=CST now that you have munged all timestamps to that TZ (you can use a [default] insteiad of host-based stanza header).
This might have to be done with a Heavy Forwarder (Index twice).

Q2:
How do I ensure that Splunk knows what TZ goes with which host so that events get timestamped correctly?
A2:
http://docs.splunk.com/Documentation/Splunk/latest/data/Applytimezoneoffsetstotimestamps

Q3:
How do I make sure that Splunk normalizes my environment so that I can specify TimePicker values in my local time and see events with times displayed in my local time?
A3:
There is a user/login-level setting that tells Splunk how to normalize timestamps when presenting data to each user. It is in Settings -> Edit Account -> Times zone. Once this is set, the TimePicker part is solved. This normalized time is shown only if you select List or Table (e.g. not Raw) in the upper-left corner control which is above the search results. Doing so adds a Time column next to the Event column showing _time normalized to your TZ for each event.

[martin_mueller]

A1 cannot work, regex replacements happen after timestamp extraction.

[woodcock]

Have you tried this (recently) or can you point to documentation to back up this claim? The reason I ask (I have not tested it) is because SEDCMD happens before indexing (that is the whole point) and because some timestamping does (can) happen later or else TZ_ALIAS could not work. The fact that it does work very heavily implies that _raw finalizes before date_zone does. If so, then A1 can work.

[martin_mueller]

Timestamp extraction happens in the merging pipeline, while regexreplacement happens after that in the typing pipeline.

https://wiki.splunk.com/Community:HowIndexingWorks

[woodcock]

So many things (exceptions) are changing in the pipeline lately (e.g. INDEXED_ETRACTIONS) that I am starting to feel like I need to re-evaluate everything that I think I know). In any case, I will take your word for it that this will for sure need a Heavy Forwarder (indexing twice) to make A1 work.

[martin_mueller]

Adding a HF will move the pipelines' processors to the HF, but the order remains the same. You'd have to cook the data twice, which is usually more trouble than it's worth.

https://answers.splunk.com/answers/224312/hf1-hf2-indexer-how-to-route-a-set-of-data-that-ha.html

Setting TZ per host is much easier than trying to modify the timestamp string per host using regex.