Getting Data In

Why a field that contains date and time becomes multivalued when TIME_FORMAT is set in props.conf (JSON)?

m_efremov
Explorer

I have data source with JSON events like this:

{"timestamp":"04-07-2015 15:57:49.726","priority":"INFO","thread":"btpool0-1294","instance":"PG1","primary":"true","category":"SplunkInteractionLogger","type":"feeext","status":"f","duration":14,"message":{"req":{"ts":"2015-07-07 15:57:49","paymentInterfaceType":"MB2.0","payerType":"MB2.0","operationType":"L","clientId":"1-8ODP6R4","providerCode":"transfer-bank","templateId":null,"amount":0.0000,"currency":"643","agreement":"3434385492","cardnumber":null,"expdate":null,"cvv2":null,"cardholderName":null,"cardid":null,"srcPointer":null,"srcPointerType":null,"billID":null,"comment":null,"providerFields":[{"providerFieldCode":"uid","type":"text","providerFieldTextValue":{"value":"M.000758477"}}]},"err":{"ts":"2015-07-07 15:57:49","code":"error-external","externalErrorCode":"N.2.24782","externalErrorMessage":"AnyText.","reqSumFlag":null,"reqSum":null,"reqSumCur":null}}}

"timestamp" field contains time of event.

My props.conf on indexers for tihs sourcetype:

[mysourcetype]
TIMESTAMP_FIELDS = "timestamp"
SHOULD_LINEMERGE = false

When I perform search at webUI, Splunk finds highlited JSON events with correctly assigned fields. But recognizes time incorrectly. It assignes "04-07-2015 15:57:49.726" to _time as 07 day of 04th month, that is incorrect in this log, because first number is day and second is month in it (but year, hours, minutes, seconds and milliseconds have the correct assignment in result value of _time).

When I modify my props.conf as described below (by adding TIME_FORMAT for correct time recognition)

[mysourcetype]
TIMESTAMP_FIELDS = "timestamp"
TIME_FORMAT = %d-%m-%Y %H:%M:%S
SHOULD_LINEMERGE = false

Splunk does not start new interpretation of timestamp as expected, but starts to interpret timestamp value as multivalue field, the first value of which is null, second value - string with time (and internal event time field becomes assinged incorrectly, splunk find null in place, specifyed by TIMESTAMP_FIELDS, and try to determine the time of event by other methods, that is usuccessfully)

Why? How to fix? I want to specify my own time format for this JSON sourcetype, for which _time is assigned by "timestamp" field value and expect that there will be one value in timestamp field. Is it possible? Are there any indirect ways if not?

0 Karma

m_efremov
Explorer

First of all, multivalued fields appear probably because there was both indexed extractions and search time extractions. I turned off search time extractions in search heads props.conf for my sourcetype:
[mysourcetype]
KV_MODE=none
and there is only one value of my timestamp field now.

But main problem still not solved. With my indexer props.conf
[mysourcetype]
TIME_FORMAT = %d-%m-%Y %H:%M:%S.%3N
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = JSON
TIMESTAMP_FIELDS = timestamp

"timestamp" field assigned by "none" value in every event.
If i comment TIME_FORMAT with "#" - no effect. Splunk indexer not recognized time into "timestamp" field of event.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try setting TIME_FORMAT = %d-%m-%Y %H:%M:%S.%3N

---
If this reply helps you, Karma would be appreciated.

m_efremov
Explorer

Already tried - no effect. Splunk support noticed me to set followTail = 0 in forwarder's inputs.conf for this sourcetype and add * to monitor stanza to monitor new and old files - I don't fully understand for what reason, but I will try and report here.

0 Karma

m_efremov
Explorer

But after some experiments and corrections of other options this option takes effect! I will post separate answer about working configurations.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...