Hi
I have a forwarder on AIX with vresion 4.3.3 that probably has a problem with its parsingqueue
I see the following in metrics.log:
02-13-2013 16:47:50.219 +0100 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=449, current_size=9, largest_size=9, smallest_size=8
02-13-2013 16:48:21.226 +0100 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=449, current_size=9, largest_size=9, smallest_size=9
splunkd.log contains a lot of :
02-13-2013 17:01:37.238 +0100 INFO TailingProcessor - ...continuing.
02-13-2013 17:01:42.241 +0100 INFO TailingProcessor - Could not send data to output queue(parsingQueue), retrying...
Restarting splunk does not change the current_size_kb or current_size values so I tried to increase the queue size following this answer:
http://splunk-base.splunk.com/answers/38218/universal-forwarder-parsingqueue-kb-size
This leads to an increase of max_size_kb and current_size_kb but does not result in the forwarder sending anything to the indexer.
If current_size indicates how many events are in the queue the this number is relatively low.
Is there a way to debug what events are stuck in a queue?
Can I somehow manually force the forwarder to empty the queue and drop the events (I know, that this is ugly)?
Another strange thing is, that once in a while (every cupple of hours) the logs are suddenly indexed, but I did not find any hints in splunkd.log or metrics.log. There is an identical system with the same configuration that works fine. The indexer is not very busy it indexes about 30-40GB a day.
Thanks for your help,
Chris
I realize this is an old thread but in case anyone is running into this, this is how I solve it:
Do a running read of splunkd.log while searching for "while reading"
tail -f /opt/splunk/var/log/splunk/splunkd.log | grep -i "while reading"
Stop splunk and keep looking at the output of the tail command. Whichever file splunk was reading while it was shutdown, is your trouble file.
Can I somehow manually force the
forwarder to empty the queue and drop
the events (I know, that this is
ugly)?
Did you find an answer for this. Thanks!
anyone, purging queue on (intermediate) forwarders stuck at 100%, without reinstalling from scratch?
Restart will by default clear the queues, if you have a specific question it may make sense to open a new Splunk answers post on it as this post is very old
Hi Chris, you just got an email 😉
If this is a forwarder, the problem is usually a step after :
Thanks for replying, the indexer queues (SOS) seem to be ok, the 256KBps is not a problem either the forwarder has a thruput close to 0 for most of the time and then from time to time indexes its data(I don't see why it behaves like this). I see a couple of WARN TcpOutputProc - Raw connection to ip=