Infoblox timezone not changing

[arlombar]

We are collecting logs from Infoblox and forwarding from the product into Splunk which is working as expected, however the timezone Splunk is indexing appears to be in GMT/UTC when the timestamps are actually in EST (when I run a search _time is 4 hours behind). I've gone through the documentation which references setting TZ in props.conf, but this has been unsuccessful so far. Also, Infoblox sends this data over a s2s tcp connection since Splunk is built in which acts as a Universal Forwarder.

Is it possible to set a TZ setting on the Infoblox side before sending logs over the Splunk or am I just missing something in my current configuration to get this to work?

For context, the infoblox DNS events do not have a timezone in the raw event and we are collecting the events in this fashion:
Infoblox (UF) -> HF -> IN

This is also my props.conf settings which are on the HF and IN:

[source::/infoblox/logs*]
TZ = US/Eastern

[host::servername*]
TZ = US/Eastern

[VatsalJagani]

@arlombar

Put below props.conf file with the proper values to it on your HF.

[source::/infoblox/logs*]
TZ = 
TIME_PREFIX = 
TIME_FORMAT = 
MAX_TIMESTAMP_LOOKAHEAD = 

Make sure your machine timings on all search heads, indexers and heavy forwarder is correct. Also, make sure your user preference timings on search heads are okay.

[arlombar]

I think you may have misread. I have a timestamp in my events, but nothing that would represent the timezone (ex. UTC-3). Just to test, I also used the above setting prior and saw no change in the event times. All of these settings seem to be controlled through Infoblox.

[VatsalJagani]

I've updated the answer. Please try this props.conf and do not forget to re-verify machine timings on all instances.

One more question why you are specifying TZ on both stanzas with host and source? Anyway on one event only one will apply.

[arlombar]

Thanks again for your suggestions. I've tried the above settings and adjusted them to what I believe would be the proper settings, but I did not see any changes in my results. The issue appears that when Infoblox's Splunk forwarder (some form of light weight forwarder not a normal UF) sends the data to my HF over s2s, I have no ability to manipulate the data in anyway. Attempting to change anything with this data in the props.conf file on my HF/IN's has had no changes at all.

The reason I have a host stanza is just for demonstration purposes of what I have tried that has no worked. I am trying to show that no settings in the props.conf file for this data source will manipulate the data in any form.

Because this is over s2s and my HF is acting as an Intermediate forwarder, is the data jumping to a queue that neither my IN or HF are able to further parse possibly? I've even tried adjusting the s2s routes to dump this source to the parsingQueue for futher data manipulation, but this has also not worked.

Also, date_zone is showing as "0" on these events and if the events were pulling the timezone from the OS I would expect to see a "local" value in this field so I am ruling that portion out.

[VatsalJagani]

Please share your sample event, props.conf and screenshot showing indexer, forwarder machine's timezone, current user timezone preference and sample search result.

[arlombar]

Sample event:
1560158368,123,Query,,11.0.0.0,12345,,X,server.name.here,EX,AAAA,1,,Y,,,,,,,,,,,,,,

This comes from a Infoblox appliance using a built in UF. I am catching this on a HF which acts almost as an intermediate forwarder, this is used because we don't want to have other products send directly to our indexers.

This is the inputs.conf file on the HF: (Again, it appears that even if I change the index or sourcetype parameters, nothing changes data wise)
[splunktcp://9990]
index = dns
sourcetype = infoblox

props.conf on HF and IN (tried in both with same config): This is my current config.
[source::/infoblox/logs*]
TZ = UTC

I also tried your settings above and these were my inputs, again, no settings I adjust are effecting the data:
[source::/infoblox/logs*]
TIME_PREFIX = ^
TIME_FORMAT = %s
MAX_TIMESTAMP_LOOKAHEAD = 11

Correct me if I'm wrong, but if the events were pulling the timezone from the "machine instance" the date_zone would show as "local", this is what is stated in the docs. My events are showing a date_zone of "0".

Current user timezone preferences are Eastern (GMT-5:00). This value doesn't matter though, for example if I set this to a UTC/GMT equivalent of the correct timezone of the timestamps, I still need to search back almost 4-5 hours to get real time events. This would indicate that the timestamps are being interpreted incorrectly at index time I would presume.

The search result is as follows:
_time is showing the timestamp in EST time and I have to search back almost 4-5 hours to get recent results so this is not a search time problem.

[VatsalJagani]

This works for me. Could you please try this?

[infoblox]
TIME_PREFIX = ^
TIME_FORMAT = %s
MAX_TIMESTAMP_LOOKAHEAD = 11
TZ = GMT

I've used sourcetype instead of source as I've doubt about the pattern that you mentioned. If you want to use source only then use ... pattern instead of *. Check props.conf for documentation about pattern.

[arlombar]

Thanks again for your suggestion, but this also did not work. I've setup a number of timezone, time prefix, and time formatting parameters before and never run into an issue like this so I'm not sure why this is happening. I have never worked with a product like this before, that uses a lightweight Splunk UF that I do not have control over and does not simply take a .conf file. For more context here i the admin guide from infoblox to show you what it looks like from their side :

https://docs.infoblox.com/display/IDCUG/Configuring+Splunk+Destination

Alot of the expected commands/parameters found on a normal UF are not available.

[oscar84x]

This is probably too simple of a suggestion but have you checked what timezone is set for the user in your Search Head's Splunk Web? This will affect the time displayed in your search results.
Check under username > preferences

[arlombar]

While changing the user preferences settings does adjust the _time value to the correct value, I still need to search back 4-5 hours just to see real-time events. I think this is an issue with the index time timezone configuration, however since Infoblox has a built-in UF, their appliance controls alot of the props.conf pieces as far as I've been able to tell, but they are limited in their settings. For example, my customer needs to set the index name and sourcetype before sending it over s2s which is a little bit different of a configuration then I've ever worked with.

[oscar84x]

So just to clarify, you updated props.conf, restarted your instances, and any new incoming data after the change is still showing the wrong time zone?
- I don't see why this would make a difference but I saw a couple of people commenting that it worked for them for the source stanza and not for the host stanza, as well as removing the spaces from the config i.e. TZ=US/Eastern.

[arlombar]

Yes on all aspects of your question. I even tried your recommendation, but am still seeing the same timezone issue.