Deployment Architecture

Splunk forwarder failed to send logs from amazon linux instance

meet_vadaria
Engager

Trying to send logs to Splunk server using forwarder installed on Amazon Linux instances. I am not seeing any data on Splunk server. On forwarder side, I am seeing interfaces.sh related error in /var/log/splunkd.log.

09-10-2019 16:04:52.161 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/xxx_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 16:04:52.171 +0000 ERROR ExecProcessor - message from 
"/opt/splunkforwarder/etc/apps/xxx_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 16:04:52.280 +0000 INFO  TailReader -   ...continuing.
09-10-2019 16:05:03.723 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxxxxx_3D3B3E31-6E53-4D7A-AB7E-0FAF1FC62062
09-10-2019 16:05:10.007 +0000 WARN  FileClassifierManager - The file '/var/log/btmp' is invalid. Reason: binary.
09-10-2019 16:05:10.007 +0000 INFO  TailReader - Ignoring file '/var/log/btmp' due to: binary
09-10-2019 16:05:16.196 +0000 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
09-10-2019 16:05:22.340 +0000 INFO  TailReader -   ...continuing.

I'm experiencing this issue with all amazon Linux servers.

0 Karma

mguhad
Communicator

Hi,
If data is being terminated before it even reaches the parsing queue indicates a connection problem as data is being dropped before it enters the first pipeline. Please check your firewalls, ports, IP TABLES etc to root out connectivity issues between the indexers and UFs/HFs.
It also worth checking the security groups of the Indexers and the UFs if they are the same or have similar permissions/rules.

Hope this helps.

0 Karma

meet_vadaria
Engager

I don't think it's a port issue. as it's able to connect to indexer on 9997 port as per logs. I also verified with telnet.

09-10-2019 20:04:21.807 +0000 INFO TcpOutputProc - Connected to idx=100.117.33.54:9997, pset=0, reuse=0. using ACK.
09-10-2019 20:04:24.586 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 20:04:24.603 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 20:05:18.036 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_100.117.8.197_8089_100.117.8.197_mongop0-i-09e03c274a86ef49b-p1-ugw1.wss.symfedcloud.com_8323AF5D-B129-41AB-8B7B-8A9E95A9C7D0
09-10-2019 20:05:24.418 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 20:05:24.419 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 20:05:31.624 +0000 INFO TcpOutputProc - Closing stream for idx=100.117.33.54:9997

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...