Getting Data In

How do we limit the length of json events?

danielbb
Motivator

In Does TRUNCATE specify the ultimate size of an event? we looked at standard logging and we are good with TRUNCATE for the max line's length and MAX_EVENTS for max number of lines. We are trying to establish limits standards for our data and we don't know now how json type data fits into these limits.

Tags (2)
0 Karma

anmolpatel
Builder

I would also utilise this search to see if there is any issues with the ingested data in terms of truncate / line breaking along with the config in props / max length in above link

 index=_internal ( warn OR error ) NOT StreamedSearch (LineBreakingProcessor OR "AggregatorMiningProcessor - Breaking event")  
        | rex "limit of (?<limit>\d+)" 
        | eval src= if(component="LineBreakingProcessor", "Line is too long - adjust truncate setting (keep eye toward line breaking)" ,"Too many lines - work on line breaking") 
        | stats count as events values(limit) as limit dc(data_source) as sources dc(data_host) as hosts by data_sourcetype src 
        | stats sum(events) as total_events list(events) as events list(src) as issue list(limit) as limit list(sources) as sources list(hosts) as hosts by data_sourcetype 
        | sort -total_events

If there are no parsing issues, your config should be good.
Note: I would revisit the config on a regular basis as there are times when there is a data feed which is outside the set limit, it just helps with refinement.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...