is forwarder working

[rhuber]

I have enabled a regular forwarder and pointed it to the receiver both of which are on my LAN. I added data input pointing to a log directory that has a variable number of application log files, the contents of which I need indexed on the receiving splunk server and enabled SplunkForwarder.

Is there something else I need to do to get the process going?

Tnx...

[LCM]

• If you're still playing around, I would suggest you to re-install splunk and let it run under root permission. Then you can be sure, there is no other issue between your indexer and forwarder.
• It that works with root, I'd go for the splunk user & group. Of course you you'd change your root-groop to splunk-group, to give it a go and do the privileges stuff

[LCM]

Funny - and if you untick (save data locally) it doesn't work?

[rhuber]

No lightforwarder, at least not yet. Changed the option to save data locally to yes. Manufactured some new data in the monitored directory and that has appeared on the receiving indexer. Good news for sure but changing the option to save data locally is the only change made.

[LCM]

I hope you don't have a lightforwarder yet. On the forwarder site, check if you storage a local copy of the datas (UI > Manager > Forwarding & Receiving > Forwarding defaults). Try on the forwarder UI if you see your datas. If that is the case, something's wrong with either the connection or the receiving site.

[rhuber]

No joy! Reinstalled splunk on the receiver and one of the forwarders as root and started as root. Set the receiver to listen on 9157. Enabled the forwarder and pointed it to the receiver on 9157, chose a directory for it to monitor. netstat -a run on both the receiver and forwarder show a connection established on port 9157 and that the receiver is listening on 9157. There are files in the directory that the forwarder is to monitor. AFAIK the receiver should be putting the data someplace under /opt/splunk/var/lib/splunk and I find nothing that appears to be forwarded data. What am i missing?

[LCM]

Sounds good - let me know if that worked

[rhuber]

Still trying to get it going or find out what it's doing. Currently I have been running as splunk and checked all the dir/files from /opt/splunk on down on the receiver all the files that
find . -ls | grep root
can find belong to splunk user and group (no output to syserr). Of the 3 follower instances 2 are have the same permissions but the third ./lib/python2.5 files belong to root/root?

Anyway I will reinstall as root and run as root to see if that makes a difference.

[LCM]

If your splunk is running as non-root user you may have to do following: http://www.splunk.com/base/Documentation/latest/Installation/RunSplunkasadifferentornon-rootuser

[rhuber]

The site says to change owner to splunk, which I did but there are files that belong to group root. Should the group membership be changed to splunk as well?

[nse]

On the receiving side, have you set up splunk for receiving? In the manager its Forwarding and receiving under System configurations.

[rhuber]

I changed all the forwarders to go to 9157 but I still do not see anything; perhaps I am looking in the wrong place. Where is splunk saving the forwarded data?

[nse]

Hmm, that is odd. Since you're having more luck with a different port have you tried forwarding on that port? If they are still not talking to each other check your splunkd.log for an error.

[rhuber]

decided to add an alternative listening port, 9157, and now netstat reports 9157 as listening but still no sign of 9997?

[rhuber]

Receiving was enabled and configured to listen on port 9997. Splunk was restarted. Manager>receiving shows port 9997 as enabled, however, the command netstat does not show tcp port 9997 as listening (in fact 9997 does not appear anywhere in the output from netstat -a. Is there a step after setting the listening port.