Events with RSYSLOG_ForwardFormat time not parsing and normalizing to UTC

[pdc_mmiling]

Hello

I have some rsyslog data coming from an rsyslog server configured with RSYSLOG_ForwardFormat to tcp port 5140 on one of the indexer cluster hosts. The data looks like this in Splunk:

<14>2016-08-21T20:36:01.770243-07:00 host01 php-fpm[] pool ......

I've attempted to parse it using a property and pushing it from the indexer master (in etc/master-apps/_cluster/local/props.conf) to the slaves:

[host::host0*]
TIME_PREFIX=<\d+>
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%6N%:z

However, the indexer does not seem to pick up on this and it thinks that the events are coming in 7 hours behind, which means no data in the 30 second window.

Can someone validate that this the correct time parser string? Is Splunk capable of parsing out the time values at index time?

Thanks

[acharlieh]

My guess is that your props.conf stanza isn't referencing the right thing, or your timestamp settings are too late.

Check out and follow along with the community wiki on how indexing works. You're using an rsyslog server to send over tcp to Splunk. In this setup Splunk is likely assigning the host by applying a regular expression to the event at index time and extracting it (this is typically done for syslog, and might even be a default extraction for your sourcetype) ... this would happen during the Typing pipeline step. Timestamp extraction however has already happened during the Merging pipeline, therefore your props.conf settings referring to the host from the event is not being picked up (as the host you're referencing is set too late).

You probably want to apply the settings referencing the host as the syslog server (or whatever is being set depending on connection_host in inputs.conf ) Or referencing the sourcetype you're using for the data.