Not getting data from Heavy Forwarder

[munisankar]

Hello,
Recently we have deployed the Splunk Enterprise.
Our moto is to monitor Wi-Fi usage, our Wi-Fi devices sending log data to syslog server, in syslog I have installed HF and configured all required settings but unfortunately am not seeing any data flow to splunk indexer.

Configuration:
Heavy Forwarder
Outputs.conf - configuration

[tcpout:group1]
server=X.X.X.X:9997
[tcpout]
indexAndForward=true

inputs.conf - configuration

[monitor:///var/log/messages]
sourcetype= cisco:ise:syslog

Splunk Enterprise
Enabled receiving in port no - 9997

inputs.conf - configuration
[default]
host = splunk server hostname
[splunktcp://9997]
disabled = 0

Firewall been adjusted not to block traffic from port.
Did ping and telnet test and both are successful but not sure why not able to see data.
kindly let me know suggestions to fix the issue.

Regards,
MC

[micahkemp]

Two likely culprits:

1) Is your forwarder sending any data to the indexer? Can you search for index=_internal host=<your forwarder> to determine if nothing is being forwarded?
1a) On your forwarder, also run splunk list forward-server to see if it's probably configured to forward.

2) Your forwarder doesn't have permission to read the logs in question. While logged in to the account the Splunk forwarder is running as, try head /var/log/messages. If you can see the lines, permissions are fine. If not, you need to figure out the linux permissions to allow that account to read the log file.

On another note, I see you have /var/log/messages set to sourcetype cisco:ise:syslog. I can't imagine that file containing data of that sourcetype, at least not primarily. But, this can be worked separately from your forwarding issue.

[munisankar]

1.Ran index=_internal host=
after running I can see thousands of events displayed in search head.

1a.Ran splunk list forward-server

No results for this query

2.Ran head /var/log/messages

I can see some lines after running this.

sourcetype is - cisco:ise:syslog bcoz cisco ise devices are configured to send data to syslog server.

[micahkemp]

1) did you fill in the host field with the name of the forwarder? And did you run this search from the search head?

1a) did you run splunk list forwarder-server on the forwarder?

2) did you run head /var/log/messages as the user that splunk is running as?

[skalliger]

Hi,

you installed a heavy forwarder on a syslog server? You may want to uninstall it and install a Universal Forwarder instead. The footprint is lighter (the other option would be to get the events via HTTP event collection). Also, specify a setting for the index; index = xyz in your inputs.conf. Otherwise, your data will go into the "main" index (which you usually don't want to).

After this, you might want to check the Unviersal Forwarder's splunkd.log for errors:

cat /opt/splunkforwarder/var/log/splunk/splunkd.log | grep ERROR

Skalli

[munisankar]

Hi,
Yes , I have installed HF in syslog server. Syslog server getting data from our wireless devices.
We want to index the data before it is reaching to indexer, I think we can't achieve this with UF.

Any suggestion to fix the issue.

Regards,
MC

[janadevops]

Have you restarted the HF after you configured the .conf files? If yes, please check the logs under /opt/splunk/var/log/splunk/splunkd.log

or cat /opt/splunkforwarder/var/log/splunk/splunkd.log | grep ERROR

you need to see the information related your UF, if not its not configured properly.

[munisankar]

I ran the command as suggested but I don't see anything related to HF.

Most of the errors are related failed authentication only.

[munisankar]

Hi p_gurav,

No am not able to search data on heavy forwarder.

Regards,
Munisankar C

[p_gurav]

Hi ,
Did you edit outputs.conf?

[munisankar]

Hi,
I have made a suggested change in outpts.conf.

I can see below error in log file:

01-11-2018 16:45:09.765 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason=' frozen_buckets'
01-11-2018 17:39:52.393 +0530 INFO TcpOutputProc - Connection to X.X.X.X:9997 closed. Connection closed by server.
01-11-2018 17:40:12.423 +0530 WARN TcpOutputProc - Cooked connection to ip= X.X.X.X:9997 timed out
01-11-2018 17:40:20.274 +0530 INFO TcpOutputProc - Connected to idx= X.X.X.X:9997

[p_gurav]

Hi munisankar,

Could you search data on heavy forwarder itself as you set indexAndForward=true?

[munisankar]

No am not able to search data in HF.

[janadevops]

how you are checking, show me the command

[munisankar]

Am trying this command in forwarder.

sourcetype="cisco:ise:syslog"

[p_gurav]

Also in outputs.conf file :
[tcpout]
defaultGroup=group1
indexAndForward=true

[mayurr98]

check errors in index=_internal

[munisankar]

Hi Mayurr98,
please let me know path where I can find the errors.

in which file I should add this index=_internal

am not sure about your last point.

[mayurr98]

login heavy forwarder and put index=_internal in search
OR else look the filepath
/opt/splunk/var/log/splunk/splunkd.log

[munisankar]

I ran it in search and getting thousands of events.

I checked syslogd.log and below are the recent information from log.

01-11-2018 16:45:09.765 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason=' frozen_buckets'
01-11-2018 17:39:52.393 +0530 INFO TcpOutputProc - Connection to X.X.X.X:9997 closed. Connection closed by server.
01-11-2018 17:40:12.423 +0530 WARN TcpOutputProc - Cooked connection to ip=40.221.2.184:9997 timed out
01-11-2018 17:40:20.274 +0530 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997

[mayurr98]

search for ERROR information