Getting Data In

Why is my table field from JSON not working on all fields?

sboogaar
Path Finder

We are working with the following JSON generated by a dcos/marathon api:

alt text

When I run:

index=dcos sourcetype="dcos:marathon:metrics" | table gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count

I get a nice table with all the expected numbers.

But, when I run:

index=dcos sourcetype="dcos:marathon:metrics" | table gauges.service.mesosphere.marathon.leaderDuration.count

All the fields are empty.

Why can I see the correct values for "gauges.api.mesosphere.marathon.core.event.impl.stream.HttpEventStreamActorMetrics.number-of-streams.count" But can not see it for gauges.service.mesosphere.marathon.leaderDuration.count
I also tried to get the data with spath like:

index=dcos sourcetype="dcos:marathon:metrics" | spath "gauges.service.mesosphere.marathon.leaderDuration.count" |  table  *

But again, the values are empty even though I can see gauges.service.mesosphere.marathon.leaderDuration.count in the table headings.

Even when I generate the searches with Splunk I get no data

alt text

Tags (2)
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Splunk has a limitation on how big a json it is able to extract.

Let's verify that is not the issue. This should snip out all the nodes in the JSON before the leaderDuration node.

 index=dcos sourcetype="dcos:marathon:metrics" 
| head 1
| rex mode=sed field=_raw "s/(gauges:\s{)(.*)(service.mesosphere.marathon.leaderDuration)/\1\3/g"

Verify that that code kills the earlier data. After that, try

| table gauges.service.mesosphere.marathon.leaderDuration.count

and

|  spath "gauges.service.mesosphere.marathon.leaderDuration.count"
0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@sboogaar
I have a doubt regarding below configurations. It might be hit in your event. Can you please reconfigure limits.conf if required and check again.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

limit = <integer>
* The maximum number of fields that an automatic key-value field extraction
  (auto kv) can generate at search time.
* If search-time field extractions are disabled (KV_MODE=none in props.conf)
  then this setting determines the number of index-time fields that will be
  returned.
* The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
  'linecount', 'splunk_server', and 'splunk_server_group' do not count against
  this limit and will always be returned.
* Increase this setting if, for example, you have indexed data with a large
  number of columns and want to ensure that searches display all fields from
  the data.
* Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

0 Karma

sboogaar
Path Finder

@kamlesh_vaghela changing the extraction_cutoff worked I did not need to update the limit. If you post it as an answer I will accept it.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Great @sboogaar, extraction_cutoff worked for you.
Glad to help you.

0 Karma

macadminrohit
Contributor

Does splunk create a field name gauges.service.mesosphere.marathon.leaderDuration.count similar to what it has created where it showed you the contents in the table.

0 Karma

sboogaar
Path Finder

@macadminrohit Yes see the last image.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@sboogaar

Can you please share the sample JSON event??

0 Karma

sboogaar
Path Finder

@kamlesh_vaghela It is 34k characters long and contains private data so I can not share it, if you tell me what you want to check I will try to provide that information.

0 Karma

woodcock
Esteemed Legend

If you cannot provide a sanitized event of identical size, then there is no good way for us to help.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...