Deployment Architecture

_introspection index error of hot bucket

ips_mandar
Builder

Hi,
I am getting below error for '_introspection' index-

The percentage of small buckets (75%) created over the last hour is high and exceeded the red thresholds (50%) for index=_introspection, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=4, small buckets=3

I want to know why this is happening with _introspection index?
I can understand if I increased hot bucket count then error may get resolved but I would like to know why it is happening?
Thanks,

0 Karma

bfeldmann_splun
Splunk Employee
Splunk Employee

@ips_mandar 

Were you ever able to confirm what was causing this issue?

0 Karma

codebuilder
SplunkTrust
SplunkTrust

If you are on Linux, the default setting in Splunk is to forward/index all files at /opt/splunk/var/log/splunk/*
This becomes an issue if you have logrotate configured, and have not updated dir monitoring.
Meaning, logrotate will either rotate log files and append a timestamp, or compress the files to .gz, or both.
Out of the box, Splunk will index all versions of the same file because it see's them as "new".

Ensure that you are not indexing locally, whitelist .log files, and blacklist everything else.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

ips_mandar
Builder

@codebuilder I am on windows os and have only batch monitoring input configured. Not sure why I am receiving older events in _introspection  index

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Have you check that you have correct time on all your nodes from where you are collecting events by UF, HF? Quite often this could mean that events from nodes comes with timestamps which are not fitting to Splunk's default time range for events in one bucket. If/when events timestamp is out of range then splunk close current host bucket after it has inserted event and create new one.
r. Ismo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...