Hi, Splunkers:
I have a forwarder that is target to a incorrect indexer and it was paused to send data for 3700s.
Now I have configured to a correct indexer URI and how can I make the forwarder restarting send the data to indexer?
You should not have to do anything. For only 3700 seconds, it should have been able to queue it and then restart where it left off when you added the correct Indexers.
In order to make the forwarder re-index the entire data. you need to clear the fishbucket. You can do this by deleting $SPLUNK_HOME/var/lib/splunk/fishbucket and restart the forwarder instance. By doing this it will make the forwarder to re-index everything. If you are looking to do this for a single file try adding CrcSalt to your inputs.conf, like crcSalt = readItAgain.
https://docs.splunk.com/Documentation/Splunk/7.3.2/Data/Monitorfilesanddirectorieswithinputs.conf
Hi aojie654,
At first check if you Splunk server is receiving logs from your target using a simple search:
index=_internal host=your_host | head 100
checking also last days or always.
If you have results there's an ingestion problem, otherwise a connection problem.
If you haven't results, try with telnet to understand if the connection is open:
telnet ip_server 9997
If ports are open to answer to your question I need of the outputs.conf of your Universal Forwarder (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).
If you have results on _internal but not other logs, you should share your inputs.conf (usually is in $SPLUNK_HOME/etc/system/local or in a dedicated App).
Ciao.
Giuseppe
Hi, Giuseppe:
I means that I was configured forwarder send data to an incorrect IP address and I was fixed it, now the forwarder could get connection with indexer but not start send data for it was been blocked. So how should I do next to enable data sending on forwarder?
Hi aojie654,
let meunderstand:
Is it correct?
Some questions or test to perform:
Ciao.
Giuseppe
Emmm...
For example,
1. I want configure forwarder forward data to 192.168.3.2:9997 but I make a mistike when edit the outputs.conf like follow:
[tcpout:jinmu]
server = 192.168.3.2:9998
Then, the following message appears in splunkd.log on forwarder:
10-16-2019 16:56:03.398 +0800 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group jinmu has been blocked for 3900 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
I fixed configuration in outputs.conf:
[tcpout:jinmu]
server = 192.168.3.2:9997
I can't recieve the forwarder data yet. (Maybe for the forwarder is blocked? )
I was restarted the forwarder after fixed the configuration file.
Hi aojie654,
did you checked the connection telnet 192.168.3.2 9997
?
did you checked if internal logs arrive to Splunk index=_internal host=your_host | head 1000
?
using CLI to restart Splunk, is there any error message?
Ciao.
Giuseppe
There is no errors occured when I restart splunk with CLI, and the other 2 forwarders is running well...
At now, I want to know did I need to wait the forwarder block time expired and no the other method to make the block time reset?
Hi aojie654,
no, you don't need the forwarder block time expires.
Telnet is ok?
if you run index=_internal host=your_host earliest=-7d latest=now | head 1000
have you results?
Ciao.
Giuseppe
Hi, Giuseppe:
It missing about 6 hours ago after I restart the forwarder.
In actually, 3 forwarders and indexer are in 4 different LAN, maybe there are some issue occures in the network of missing forwarder.
for this reason I asked the two checks
what are the checks results?
ciao.
Giuseppe