Getting Data In

Automatic extraction of JSON fields doesn't work and have to use spath

Motoko89
Path Finder

I have a distributed Splunk deployment and need to index JSON data, 1 object per row. Objects are serialized using NewtonSoft.JSON .NET library. I already configured both INDEXED_EXTRACTIONS = json and KV_MODE = json for my custom source type in props.conf for deployment app of forwarders, indexers, and heads.

Yet, when I search, I still need to specify spath else no result is returned. Our event text length does not exceed 5000 chars (~ 2500 chars) so I don't think the problem here https://answers.splunk.com/answers/177410/why-are-only-some-json-fields-extracted-as-data-in.html affect us. What do I miss here? Any suggestion?

My props.conf

[myblob]
DATETIME_CONFIG = 
INDEXED_EXTRACTIONS = json
KV_MODE = json
AUTO_KV_JSON = true
NO_BINARY_CHECK = true
category = Structured
description = my blob description
disabled = false
TIME_PREFIX=\"EventInfo\.Time\":\s*
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%7NZ
0 Karma

ssadanala1
Contributor

You cant specify both INDEXED_EXTRACTIONS and KV_MODE in the props .
You need to specify either of one .

Try this

[myblob]
DATETIME_CONFIG =
KV_MODE = json
AUTO_KV_JSON = true
NO_BINARY_CHECK = true
category = Structured
description = my blob description
disabled = false
TIME_PREFIX=\"EventInfo.Time\":\s*
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%7NZ

0 Karma

Motoko89
Path Finder

So there are 3 places the props.conf go to: the forwarder, the indexer, and the head. Could you advice more specifically which one I should change into what? Do I keep AUTO_KV_JSON at all 3 places?

0 Karma

MuS
Legend

AUTO_KV_JSON is not needed anywhere, because it defaults to true anyway 😉

AUTO_KV_JSON = [true|false]
* Used for search-time field extractions only.
* Specifies whether to try json extraction automatically.
* Defaults to true.

If you use KV_MODE=json it will only go on the search head. And adding to this, your JSON format must be true and valid JSON otherwise Splunk will not do anything ...

* The 'xml' and 'json' modes will not extract any fields when used on data
  that isn't of the correct format (JSON or XML).

Hope this helps ...

cheers, MuS

xpac
SplunkTrust
SplunkTrust

...and you most likely want to use KV_MODE and not indexed extractions. Remember that in case 9d KV_MODE, this has to go on the search head, not on the indexer. In case of indexed extractions, it has to go on the indexer, and maybe on the search head (not sure about that).

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...