Hi,
Last date when data got indexed was on Sep 7, 2014 09:00:00 AM. After that indexing stopped.
As per splunkd.log of our indexer,
"INFO TcpInputProc - Stopping IPv4 port" - logged twice(14:18 and 15:14) for the ports 30534, 30544, 30554 (hourly once) in splunkd.log on 5th Sep.
"INFO TcpInputProc - Stopping IPv4 port" - logged zero times on 6th Sep 2014 ( Saturday )
"INFO TcpInputProc - Stopping IPv4 port" - logged 30 times on 7th Sep 2014 ( Sunday ) - ie once for each port at 23:26
"WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds" - logged once only on 7th Sep 2014 ( Sunday ) at 23:26
"ERROR TcpOutputFd - Connection to host= failed" was logged from 09-07-2014 23:26 to 09-07-2014 23:57 for 15 times.
Didn't see this error on 5th or 6th Sep 2014.
On searching for blocked logs in metrics.log of our indexer,
Nil on 5th Sep, 6th Sep. Sep 7th 2:40 to Sep 8 15:04 --> 3459 times blocked=true
On further analysing, indexqueue, splunktcpin, typingqueue and aggqueue were found to be blocked as follows :
09-07-2014 18:44:03.649 +0100 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=2493, largest_size=2504, smallest_size=1206
09-07-2014 21:08:44.626 +0100 INFO Metrics - group=queue, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=2508, largest_size=2517, smallest_size=323
....
09-07-2014 23:59:21.983 +0100 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=2310, largest_size=2310, smallest_size=2310
09-07-2014 21:08:44.626 +0100 INFO Metrics - group=queue, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=2508, largest_size=2517, smallest_size=323
....
09-08-2014 00:06:36.011 +0100 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1432, largest_size=1432, smallest_size=1410
09-08-2014 01:51:55.416 +0100 INFO Metrics - group=queue, name=aggqueue, blocked=true, max_size_kb=1024, current_size_kb=1023, current_size=2984, largest_size=2984, smallest_size=2952
Please suggest.
Thanks,
Seema
... View more