I have data source with JSON events like this:
{"timestamp":"04-07-2015 15:57:49.726","priority":"INFO","thread":"btpool0-1294","instance":"PG1","primary":"true","category":"SplunkInteractionLogger","type":"feeext","status":"f","duration":14,"message":{"req":{"ts":"2015-07-07 15:57:49","paymentInterfaceType":"MB2.0","payerType":"MB2.0","operationType":"L","clientId":"1-8ODP6R4","providerCode":"transfer-bank","templateId":null,"amount":0.0000,"currency":"643","agreement":"3434385492","cardnumber":null,"expdate":null,"cvv2":null,"cardholderName":null,"cardid":null,"srcPointer":null,"srcPointerType":null,"billID":null,"comment":null,"providerFields":[{"providerFieldCode":"uid","type":"text","providerFieldTextValue":{"value":"M.000758477"}}]},"err":{"ts":"2015-07-07 15:57:49","code":"error-external","externalErrorCode":"N.2.24782","externalErrorMessage":"AnyText.","reqSumFlag":null,"reqSum":null,"reqSumCur":null}}}
"timestamp" field contains time of event.
My props.conf on indexers for tihs sourcetype:
[mysourcetype]
TIMESTAMP_FIELDS = "timestamp"
SHOULD_LINEMERGE = false
When I perform search at webUI, Splunk finds highlited JSON events with correctly assigned fields. But recognizes time incorrectly. It assignes "04-07-2015 15:57:49.726" to _time as 07 day of 04th month, that is incorrect in this log, because first number is day and second is month in it (but year, hours, minutes, seconds and milliseconds have the correct assignment in result value of _time).
When I modify my props.conf as described below (by adding TIME_FORMAT for correct time recognition)
[mysourcetype]
TIMESTAMP_FIELDS = "timestamp"
TIME_FORMAT = %d-%m-%Y %H:%M:%S
SHOULD_LINEMERGE = false
Splunk does not start new interpretation of timestamp as expected, but starts to interpret timestamp value as multivalue field, the first value of which is null, second value - string with time (and internal event time field becomes assinged incorrectly, splunk find null in place, specifyed by TIMESTAMP_FIELDS, and try to determine the time of event by other methods, that is usuccessfully)
Why? How to fix? I want to specify my own time format for this JSON sourcetype, for which _time is assigned by "timestamp" field value and expect that there will be one value in timestamp field. Is it possible? Are there any indirect ways if not?
First of all, multivalued fields appear probably because there was both indexed extractions and search time extractions. I turned off search time extractions in search heads props.conf for my sourcetype:
[mysourcetype]
KV_MODE=none
and there is only one value of my timestamp field now.
But main problem still not solved. With my indexer props.conf
[mysourcetype]
TIME_FORMAT = %d-%m-%Y %H:%M:%S.%3N
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = JSON
TIMESTAMP_FIELDS = timestamp
"timestamp" field assigned by "none" value in every event.
If i comment TIME_FORMAT with "#" - no effect. Splunk indexer not recognized time into "timestamp" field of event.
Try setting TIME_FORMAT = %d-%m-%Y %H:%M:%S.%3N
Already tried - no effect. Splunk support noticed me to set followTail = 0 in forwarder's inputs.conf for this sourcetype and add * to monitor stanza to monitor new and old files - I don't fully understand for what reason, but I will try and report here.
But after some experiments and corrections of other options this option takes effect! I will post separate answer about working configurations.