We got a license warning yesterday and we are pretty sure it's due to excessive DEBUG events coming through. Is it possible to create a report specifying the top current indexes with DEBUG events?
Is there a way to intercept the DEBUG events at parsing time and discard them?
To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.
To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.
To answer your second question first, you can use method mentioned in below link to discard a specific event and index the rest. This needs to be set on Indexer/heavy forwarder whichever comes first in the data flow.
To identify indexes which have DEBUG events, you need to identify a pattern/rule for it. E.g. the data includes a field call log_level or loglevel with value as DEBUG, OR the raw data contains keyword "debug:" or similar. The same pattern/regular expression can be used to discard them.
Great. Normally the events contain the word DEBUG in upper-case. Just based on that, can we create a query which would give a break-up of today's DEBUG data by the indexes?
index=* DEBUG | stats count by index | sort - count
is not bad - how can I enforce only upper-case DEBUG?
@somesoni2 is right this search will be a heavy hit in terms of performance so make sure to filter by time so that you are only searching what you have not already checked. Also, I would bet that this only occurs in specific indexes and sourcetypes. Filter by just those indexes and sourcetypes as well.
Right right Claw - scary to run it in production, which I'm doing now ; - )
Well, it'll be long *** query.
index=* CASE(DEBUG) | stats count by index
or useful but even worst in terms of performance.
index=* | eval isDebug=if(searchmatch("DEBUG"),1,0) | stats count as Total sum(isDebug) as Debug by index | eval Perc=Debug*100/Total
Wow - gorgeous