Hello
I had installed a new test setup on a server to index some rolling log files, i know that it is defiantly not 500Mb per day of logs.
also the below violations from Main indexer, i think by default Files & directories indexing is stored in internal Index right?
I had received some python errors during installation, but i ignored them
Thanks
Sandy
=================================================================================================
Index name Max size (MB) of entire index Frozen archive path Current size (in MB) Event count Earliest event Latest event Home path App Status Actions
main 500,000 N/A 16,501 432,673,653 Sep 22, 2011 10:35:03 AM Sep 15, 2014 1:34:26 PM C:\Program Files\Splunk\var\lib\splunk\defaultdb\db system Enabled | Disable
=================================================================================================
Severity Time Message Indexer Pool Stack Category
This pool is over poolsz=524288000 bytes, please correct before midnight auto_generated_pool_download-trial download-trial pool_over_quota
This pool contains 1 slave/s in violation auto_generated_pool_download-trial download-trial pool_violated_slave_count
This pool contains slave(s) with 9 warnings Vin2 auto_generated_pool_download-trial download-trial pool_warning_count
=================================================================================================
Severity Time Message Indexer Pool Stack Category
Sep 15, 2014 12:00:00 AM
(13 hours ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 14, 2014 12:00:00 AM
(1 day ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 13, 2014 12:00:00 AM
(2 days ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 12, 2014 12:00:00 AM
(3 days ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 11, 2014 12:00:00 AM
(4 days ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 10, 2014 12:00:00 AM
(5 days ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 9, 2014 12:00:00 AM
(6 days ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 8, 2014 12:00:00 AM
(1 week ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
Sep 7, 2014 12:00:00 AM
(1 week ago) This pool has exceeded its configured poolsize=524288000 bytes. A warning has been recorded for all members Vin2 auto_generated_pool_download-trial download-trial license_window
=================================================================================================
I see rate of 60KB/s
and looks like some issue how it is reading the files i have 10 log files which rotate after every 1MB it takes about an 2-3 hour to rotate them and roll over, so i should have 200 Mb or less, may be i will rebuild from scratch and try again as this set up is pretty much useless for next 30 days
configlog.0
configlog.1
configlog.2
configlog.9
Thanks
60KB/s works out to 5GB/day. Use the split-by fields to track who's to blame.
So on average you were indexing a couple of gigabytes, unless those were all within a day it's no surprise you've exceeded the 500MB quota often.
In the SoS app go to to Indexing -> Indexing Performance and set the time to last 30 days. You'll see what sourcetype, index, host, or source has delivered how much data to your indexer and on what day. My guess is the number of license warnings will line up with the number of days your indexing performance shows over 500MB/day. Note it shows KB/s, so you're looking for about 6KB/s to make 500MB/d on average.
Thanks martin,
This was installed 10 days ago, sep 5th
i have installed SOS app , it has lot of info what should i be looking for?
i don't even have slave configuration it shows
Slave Warning Information
Slave GUID Pool Hard Warnings In Violation?
Vision2 09FF76F9-A022-4DDF auto_generated_pool_download-trial 9 yes
Your main index is 16.5GB in size, which suggests maybe 30-50GB of indexing volume, depending on the type of data. For how long has this been running?
To better diagnose your issue, look at the license usage report if you're on Splunk 6 or grab the SoS app if you're not. Actually, grab that anyway - can't hurt.