Can I change the stats limit in Splunk for the max characters?

[Hugues]

hello all,

My problem is I thing Splunk have max character accepted for stats command,

when i perform this search

index="bnc_6261_pr_log_conf"  logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"

I see 3 event, and now if I perform this request

index="bnc_6261_pr_log_conf"  logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | stats values(l) as NumberOfCar

I received two len, one was lose

172

6277

if I perform this statistic request :

index="bnc_6261_pr_log_conf" | logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

recived:

Max(Length):29886

perc95(Length):275756

The event I lose have effectively 28973 character,  I thing the actual limit is 10 000.... I already change TRUNCATE parameter at 80 000. It for that I can load event with up to 10 000 character....

My question is, Can I change the stats  limit in splunk for the max characters ? with which parameter ? and where from the web page ? can be change by non admin  and for a  specific source ?

Thank for your future help.

Hugues

[Hugues]

Hello all, for some reason I need pass trought _raw my message before use stats.

index="bnc_6261_pr_log_conf" logStreamName="*/i-*/config_Ec2_CECIO_Linux/stdout"

| sort 0 _time

| rex field=_raw ".*message\":(?<new_message>.*)"                        (add my message from _raw in new_message)

| rex mode=sed field=new_message "s/\\\n/\n/g"                          (I lose my asci format, bring back newline)

| stats list(new_message) as msg by logStreamName Machine_name account

with the _raw , I keep all my events with stats command. Some expert from Splunk , start check why I need to do that ... If they find the raison I will communicate the info with you...

[johnhuang]

For Auto KV (key valued) extraction, Splunk has a default limit of 10240 characters. You can increase that limit by adjusting maxchars value in limits.conf.

maxchars = <integer>
* When non-zero, truncate _raw to this size and then do auto KV.
* Default: 10240 characters

source: https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Limitsconf

[Hugues]

Thanks , 

 it for all app index or it can be only apply on index="bnc_6261_pr_log_conf" ? im a newbie in splunk and  im not sure if I use the right word... This change will effect anybody else in our compagnie  all other search on other index?

thanks again

Hugues

[richgalloway]

The maxchars setting applies to all indexes.

[PickleRick]

Your thinking is wrong.

Check this example:

| makeresults 
| eval a="a,a,a" 
| eval a=split(a,",")
| mvexpand a
| stats values(a)

It generates three different rows, all containing field a with value "a" but in the end stats values(a) returns only one "a", because it lists only unique values. So if any two of your three events have the same length of message field, you'll get just two results at the end of your stats.

And it has nothing to do with _raw message, or anything else so your further "analysis" is pointless.

Oh, and don't use wildcards at the beginning of search terms. They are extremely inefficient.

[richgalloway]

---

@Hugues wrote:

hello all,

My problem is I thing Splunk have max character accepted for stats command,

when i perform this search

index="bnc_6261_pr_log_conf"  logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"

I see 3 event, and now if I perform this request

index="bnc_6261_pr_log_conf"  logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | stats values(l) as NumberOfCar

I received two len, one was lose

172

6277

We don't have enough information to say this is a problem.  If two message fields have the same length then only two values will be displayed.  Use stats list(l) to view all lengths rather than just the unique ones.

if I perform this statistic request :

index="bnc_6261_pr_log_conf" | logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

recived:

Max(Length):29886

perc95(Length):275756

The event I lose have effectively 28973 character,  I thing the actual limit is 10 000.... I already change TRUNCATE parameter at 80 000. It for that I can load event with up to 10 000 character....

You're comparing apples to oranges.  The previous query gets the length (only) of the message field whereas this query gets the length of the entire event.

I don't see evidence of event truncation, but if it is happening then there will be messages in splunkd.log saying so.  To find them use this search 

index=_internal sourcetype=splunkd component=linebreakingprocessor message="truncating*"

---

My question is, Can I change the stats  limit in splunk for the max characters ? with which parameter ? and where from the web page ? can be change by non admin  and for a  specific source ?

The stats command does not have a character limit.  Some stats functions have a limit on the number of results they can return, but that does not appear to apply here.

[Hugues]

Thanks for your reply, Sorry im a newbie , I try give you much detail possible... 

I try give you more information. When I try this request,

>index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"

Output of 3 event with different message content and lenght:

Event 11:31:51/ 

Event 11:31:16:

Event 11:30:46/

If I count of length of each message, I have only two length in the output, The biggest message count are not there. All message are different , time is different and all 3 have the same logstream name.

>index="bnc_6261_pr_log_conf"  logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" | eval l = len(message) | stats values(l) as NumberOfCar

I create this small exemple  for try to explain my problem. I have same problem with a more complexe request. 

>index="bnc_6261_pr_log_conf" logStreamName="c36be289-86f8-4406-94ea-6933b26f9767/i-0ee87265c1f2f4fb7/PatchLinux/stdout" | stats list(message) as msg by logStreamName

my list msg for each logstram name lose the biggest message when is bigger of some number of caracters, may be 10 000.

Il try your request ,

>index=_internal sourcetype=splunkd component=linebreakingprocessor message="truncating*"

and this one

>index=_internal sourcetype=splunkd component=linebreakingprocessor

and 

>index=_internal sourcetype=splunkd

I have no events

Thanks all ,for your help!

Hugues

[richgalloway]

The screenshots show odd behavior.  Please run this query to see what Splunk is setting message lengths to.

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" 
| eval l = len(message) 
| table l message

That no events are returned from index=_internal sourcetype=splunkd means you don't have access to that index.  An admin can run that query for you to determine if there are any event truncations.

[Hugues]

with raw option it is work

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"
| rex field=_raw ".*message\":(?<new_message>.*)"
| eval l = len(new_message)
| stats list(l) as NumberOfCar

give this output

Now how can solve the problem with this request

index="bnc_6261_pr_log_conf" logStreamName="c36be289-86f8-4406-94ea-6933b26f9767/i-0ee87265c1f2f4fb7/PatchLinux/stdout" |  stats list(ms) as msg by logStreamName

Hugues

[richgalloway]

Glad you got the first request working.  What's the problem with the second one?  Does the ms field exist?

[Hugues]

yes it is working if I use a raw option with a regex, all events have message field, This link with the number of characters , some max.

I cannot apply this solution (ex field=_raw ".*message\":(?<new_message>.*)")  on this more complexe request

index="bnc_6261_pr_log_conf" logStreamName="*b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout"   |  stats list(ms) as msg by logStreamName

   because I have the same issue I lose the event have message with more 10 000 characters...

Hugues

[richgalloway]

The stats list(foo) command requires the foo field to be extracted beforehand, either automatically or as part of the current query.  That's why the query started working after the rex command was added to the previous query - it ensured the message field was present in all events.

[Hugues]

All 3 messages have the field message, Check the screenshot I made ... 

and I lose the message 11:31:51... 

see screenshot

when I perform this request or other type of request with eval on field message , I lose event 11:31:51 in the list msg, only 2 msg appear in List msg. 

if I try easy request index="bnc_6261_pr_log_conf" logStreamName="4a0dc28c-98ce-4f60-b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" AND message="*LONGOS*" , this suppose get out only the message 11:31:51, I get nothing ...

The Event:

output of my search on logStreamName="4a0dc28c-98ce-4f60-b6b3-f8d14815eaf8/i-09bfc06d1ff10cb79/config_Ec2_CECIO_Linux/stdout" AND message="*LONGOS*"

Hugues

[johnhuang]

Check my response on maxchars on KV extraction. It looks like your fields are extracted using KV.

[Hugues]

Thanks for your help.

only two event output

not see the event for 11:31:56 with 28973 characters

I have many other exemple with this problem.... It is not the only case.... every time I have a message with more 6000 characters, may be 10 000 , it disapear when I use stats , eval and table command....

thanks for your help. I am a newbie in splunk , your help is precious!

Hugues