Splunk Search

Is there a limit on searchable characters in an event?

t_splunk_d
Path Finder

I am searching on an event with has on an average 25000 - 30000 characters. When I search on the auto extracted fields or regex extracted fields I do not get results for the field value as it is not matching. I think there is a limit for search on the events.

For example:

index=test status=true is returning incorrect match results.
It returns the events if status=true is within first 10000 characters of the event otherwise it does not.

Is there a limit and how this can be changed? Any index specific change or any search keyword can overcome this?

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi t_splunk_d,

Update after feedback and some more research:

This is a default setting in limits.conf related to the automatic kay value extraction of _raw

maxchars = <integer>
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

After increasing this to a higher number I was able to use KV pairs after 10000 characters.

Another reason could be the event truncation described below.

this is not limit in the search, your data was truncated by Splunk.
Splunk truncates by default events after 10000 bytes or characters, see the docs http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf for more details but here is the important part:

TRUNCATE = <non-negative integer>
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
  otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often
  a sign of garbage data).
* Defaults to 10000 bytes.

to change this, you need to set in props.conf a high truncate value for the source or sourcetype:

 [YourSourceTypeHere]
 TRUNCATE = a higher number than the maximum length of your events

apply this on the parsing instance of Splunk (index or heavy weight forwarder), restart this instance and any new data will no longer be truncated.

Hope this helps ...

cheers, MuS

View solution in original post

t_splunk_d
Path Finder

Experts - Any thoughts?
The event is sometimes 20000 or more and appears in the search. But field values does not shows up (if it is above the 10000) or truncates ( when in the 10000 borderline). Is this limitation of splunk?

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi t_splunk_d, see my updated answer - Can I call myself Expert now 🙂

0 Karma

t_splunk_d
Path Finder

Yes!! No doubt you are an Expert! Thank you!

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi t_splunk_d,

Update after feedback and some more research:

This is a default setting in limits.conf related to the automatic kay value extraction of _raw

maxchars = <integer>
* Truncate _raw to this size and then do auto KV.
* Default: 10240 characters

After increasing this to a higher number I was able to use KV pairs after 10000 characters.

Another reason could be the event truncation described below.

this is not limit in the search, your data was truncated by Splunk.
Splunk truncates by default events after 10000 bytes or characters, see the docs http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf for more details but here is the important part:

TRUNCATE = <non-negative integer>
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
  otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often
  a sign of garbage data).
* Defaults to 10000 bytes.

to change this, you need to set in props.conf a high truncate value for the source or sourcetype:

 [YourSourceTypeHere]
 TRUNCATE = a higher number than the maximum length of your events

apply this on the parsing instance of Splunk (index or heavy weight forwarder), restart this instance and any new data will no longer be truncated.

Hope this helps ...

cheers, MuS

t_splunk_d
Path Finder

It is not the truncation of event, the field/keyword is present in the event but not searchable.
Is there a limitation to search for a keyword if it is located beyond 10000? I consistently see that in my search results that if I search for a field which is located beyond 10000 it is not able to locate it, whereas if the same filed is located before 10000 then it is able to locate. I am sure that there is limitation because when I search for the keyword/field the value is returned truncated. Whereas when i copy the whole event into an editor and search the keyword/field it is present. I see this truncation/no results if the keyword/field values are located beyond 10000.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

what version of splunk are you on?

0 Karma

t_splunk_d
Path Finder

Splunk Enterprise 6.5.2

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...