Getting Data In

Using indexer discovery, how to check if a forwarder has forwarded a file to the indexer cluster?

guotao4321
Path Finder

Issue:
- After uploading file to forwarder monitoring directory, we cannot search it on search head.
Environment:
- 1 search head --> 1 indexer cluster {1 master + 3 indexers} <-- 1 universal forwarder
- enable "Forward master node data to the indexer layer": http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Forwardmasterdata
- configure "Use indexer discovery to connect forwarders to peer nodes": http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/indexerdiscovery

splunkd.log on Forwarder:

11-24-2016 11:07:24.347 +0800 INFO TcpOutputProc - Closing stream for idx=172.16.1.81:9997
11-24-2016 11:07:24.348 +0800 INFO TcpOutputProc - Connected to idx=172.16.1.82:9997 using ACK.
11-24-2016 11:07:38.544 +0800 INFO TailReader - Archive file='/data/tutorialdata.zip' updated less than 10000ms ago, will not read it until it stops changing. File size=0
11-24-2016 11:07:48.598 +0800 INFO TailReader - Archive file='/data/tutorialdata.zip' has stopped changing, will read it now.
11-24-2016 11:07:48.598 +0800 INFO ArchiveProcessor - Handling file=/data/tutorialdata.zip
11-24-2016 11:07:48.598 +0800 INFO ArchiveProcessor - new tailer already processed path=/data/tutorialdata.zip
11-24-2016 11:07:54.207 +0800 INFO TcpOutputProc - Closing stream for idx=172.16.1.82:9997

11-24-2016 11:07:54.207 +0800 INFO TcpOutputProc - Connected to idx=172.16.1.81:9997 using ACK.

Findings:
1. the forwarder has already handled the file. How can we check if it successfully forwards it to the indexer cluster?
2. the forwarder is continuing to change the connected indexers. Is it normal or an issue of the communication between the forwarder and indexers?

Thank you very much for helps.

0 Karma

lguinn2
Legend

The forwarder will continue to change the connected indexer. That is called "auto load balancing" and it is the desired behavior. It is also the default.

If you want to know if the file has arrived on the indexer, you only need to search for it:

index=* source="/data/tutorialdata.zip"

If the file does not appear when you search, check to see what index was used in the inputs.conf on the forwarder. Make sure that index exists on the indexers and that you have permission to read it.

0 Karma

guotao4321
Path Finder

Thanks for the reply. Glad to know that changing connected indexer is a normal behavior, so it's easy to troubleshoot this issue. We tried other file price.csv.zip and run the search * source="/data/price.csv.zip". IT WORKS. Therefore we think it is the issue about the file.

Actually, when we create the indexers in the cluster, we clone a previous distrubuted index where we had forwarded the tutorialdata.zip. Although we remove all the database on the new indexer, will it save the hash or something else to mark the file forwarded? When we forward tutuorialdata.zip again, the indexer will ignore it by checking the hash?

If it is, how can we clean the hash records to make the new indexer working for a duplicate file?

Thank you very much.

Regards,
Tao

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...