We are working with the following JSON generated by a dcos/marathon api:
When I run:
index=dcos sourcetype="dcos:marathon:metrics" | table gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count
I get a nice table with all the expected numbers.
But, when I run:
index=dcos sourcetype="dcos:marathon:metrics" | table gauges.service.mesosphere.marathon.leaderDuration.count
All the fields are empty.
Why can I see the correct values for "gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count" But can not see it for gauges.service.mesosphere.marathon.leaderDuration.count
I also tried to get the data with spath like:
index=dcos sourcetype="dcos:marathon:metrics" | spath "gauges.service.mesosphere.marathon.leaderDuration.count" | table *
But again, the values are empty even though I can see gauges.service.mesosphere.marathon.leaderDuration.count in the table headings.
Even when I generate the searches with Splunk I get no data
Splunk has a limitation on how big a json it is able to extract.
Let's verify that is not the issue. This should snip out all the nodes in the JSON before the leaderDuration node.
index=dcos sourcetype="dcos:marathon:metrics"
| head 1
| rex mode=sed field=_raw "s/(gauges:\s{)(.*)(service.mesosphere.marathon.leaderDuration)/\1\3/g"
Verify that that code kills the earlier data. After that, try
| table gauges.service.mesosphere.marathon.leaderDuration.count
and
| spath "gauges.service.mesosphere.marathon.leaderDuration.count"
@sboogaar
I have a doubt regarding below configurations. It might be hit in your event. Can you please reconfigure limits.conf if required and check again.
extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
<integer> number of bytes.
* Default: 5000
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D
limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
(auto kv) can generate at search time.
* If search-time field extractions are disabled (KV_MODE=none in props.conf)
then this setting determines the number of index-time fields that will be
returned.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
'linecount', 'splunk_server', and 'splunk_server_group' do not count against
this limit and will always be returned.
* Increase this setting if, for example, you have indexed data with a large
number of columns and want to ensure that searches display all fields from
the data.
* Default: 100
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D
@kamlesh_vaghela changing the extraction_cutoff worked I did not need to update the limit. If you post it as an answer I will accept it.
Great @sboogaar, extraction_cutoff worked for you.
Glad to help you.
Does splunk create a field name gauges.service.mesosphere.marathon.leaderDuration.count
similar to what it has created where it showed you the contents in the table.
@macadminrohit Yes see the last image.
@sboogaar
Can you please share the sample JSON event??
@kamlesh_vaghela It is 34k characters long and contains private data so I can not share it, if you tell me what you want to check I will try to provide that information.
If you cannot provide a sanitized event of identical size, then there is no good way for us to help.