Timezone Query

[willadams]

I have a Deploy server application that I use to control my "SYSLOG" server that receives logs from various other sources. The SYSLOG server has a SPLUNK UF installed on it and it sends the data to configured indexes with the relevant source types. I have a range of data sources in this app to direct where my data goes. The UF is effectively monitoring for files in a directory structure. For example

[monitor:///data/splunkforwarder/myfiles/app1/*/messages*]
host_segment = 5
sourcetype = app1_sourcetype
index = app1

[monitor:///data/splunkforwarder/myfiles/app2/*/messages*]
host_segment = 5
sourcetype = app2_sourcetype
index = app2

I have a monitor input that is using the standard JSON provided by SPLUNK for another directory

[monitor:///data/splunkforwarder/myfiles/BIGAPP/*/messages*]
host_segment = 5
sourcetype = json
index = bigapp

BIGAPP sends its logs via SYSLOG and this works as expected, however the time that is indexed in SPLUNK is out by 8 hours. The event arrives at say 8:45pm but SPLUNK indexes this at 12:45 (difference of 8 hours). I attempted to do the following:

[monitor:///data/splunkforwarder/myfiles/BIGAPP/*/messages*]
host_segment = 5
sourcetype = json
index = bigapp
TZ = Australia/Perth

I reloaded my DS and resent a log but this made no difference. From reading the articles, it would seem to indicate that this must only be done in props.conf? Do I have to create a new sourcetype (effectively duplicating the JSON sourcetype) and then apply this props to my SYSLOG application?

I don't want impact my app as all of the other monitored files are accurate from a time stamp perspective so I only need to change this one. The BIGAPP vendor does not have support for changing the time zone on the syslog so I have to resort to having SPLUNK fix this.

Thanks for any assistance.

[manjunathmeti]

1. Check the timezone of timestamp field (_time extracted from data) values in your BIGAPP json data. If values contains timezone: UTC or +00.00 or -00.00 then indexing happening correctly. You just need to change your time zone settings in Splunk search heads or search the data with keeping UTC timezone in mind.

2. If timezone in the BIGAPP data is wrong then TZ attribute won't help you as splunk docs says:

TZ = <**timezone identifier**>
The algorithm for determining the time zone for a particular event is as follows:
  * If the event has a timezone in its raw text (for example, UTC, -08:00), use that.
  * If TZ is set to a valid timezone string, use that.
  * If the event was forwarded, and the forwarder-indexer connection uses the version 6.0 and higher forwarding protocol, use the timezone provided by the forwarder.
  * Otherwise, use the timezone of the system that is running splunkd.

In this case you can set TIME_PREFIX and TIME_FORMAT in props.conf.

[new_sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE = none
TIME_PREFIX = fieldname:
TIME_FORMAT = %y/%m/%d %H:%M:%S.%3N

3. If timezone does not exist in BIGAPP data then you can set TZ directly.

[new_sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE = none
TZ = Australia/Perth

[willadams]

The indexing is correct but the time is wrong. As I mentioned all of my other data is accurate so I am not changing the timezone on my search heads for 1 source. I would like to get this particular sourcetype (only this one) and have it indexed with the correct timezone. So it would appear that I need to create my own sourcetype and configure it with the right time zone information?

[manjunathmeti]

Does timezone exist in the BIGAPP data? If yes, is it correct one?

[manjunathmeti]

If yes then refer to point number 3 else point number 4 in answer.

[willadams]

Apologies for not getting back sooner, have had other priorities. Anyway, I ran my search as follows:

index=bigapp | head 2 | fields _time

The time field shows as:

2020-04-02T03:33:08.100+08:00

So it looks like it is picking up the timezone correctly at least from the log perspective. The ingest time is therefore 03:33:08. My Search Head is configured with my timezone and all my other data is showing as current time (+8). I can't change the local source (at the moment), is there any way I can make SPLUNK add +8? I think the answer will be not recommended and to either run searches and adding in +8 or have my SYSLOG instance modify the time at time of receipt.

[willadams]

So if I set my search time to last 60 minutes, I cannot find the events. If I then set my time to be "8 hours ago" I will see my events. Is there a way I can drive SPLUNK to show this time appropriately. The device that sends the initial syslog does not have an option to change the log format timezone so it will always show utc+0