Getting Data In

Splunk not recording events on hosts

pfcnetwork
New Member

Hello,

I am running the free version of Splunk for syslog (v 5.0.3) and after a server reboot last week, it is no longer recording events for any of my hosts that I can see in the search index (I have 10 or so).

I have tried restarting the splunkd and splunkweb services, but no luck.

Any suggestions would be approeciated

Cheers

Tags (1)
0 Karma

linu1988
Champion

Free version how many days have passed? is it like you are now not able to search them?

0 Karma

lukejadamec
Super Champion

Here is a little Splunk magic that might work....

Put this stanza in the etc/system/local/inputs.conf file on the Indexer, and restart splunkd.

[splunktcp://9997]
Connection_host = none

0 Karma

pfcnetwork
New Member

Our hosts are either on the same LAN or connected via site-to-site IPSec VPN which forwards all traffic through the FW; it's not checked against the access lists.

0 Karma

lukejadamec
Super Champion

Do you have a firewall blocking traffic on UDP514?
Is the input still configured for the syslog input?

0 Karma

pfcnetwork
New Member

UDP port 514 is not in use.

0 Karma

lukejadamec
Super Champion

If these are syslog inputs, then on the indexer you might see errors in the splunkd log. If they are syslog inputs then you probably don't have forwarders installed on the hosts that are not reporting. If all of the hosts went offline at the same time, there are no errors in the splunkd log, and you still have the syslog input active, then it sounds like a problem with the syslog port on your indexer UDP 514. From a command line run netstat and look for port 514 to see if it is in use.

0 Karma

pfcnetwork
New Member

Sorry - I'm rather new to Splunk and not sure what you are referring to. There's only a few lines in the log file with 'forwarder' in them and they all read like this:

02-25-2014 14:37:19.985 -0700 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder

0 Karma

lukejadamec
Super Champion

This is a syslog input? Is the port in use by something else?

0 Karma

linu1988
Champion

please go and check in forwarder splunkd.log why it's not forwarding rather than the search head

0 Karma

pfcnetwork
New Member

On the main Summary page where you can select a host from the 'Hosts' list. Or doing a 'host=name/IP' search yields nothing beyond Feb 18th.

In the Splunkd log file, what exactly should I be looking for? There's nothing that explictly says 'error'

0 Karma

lukejadamec
Super Champion

Are you sure the data is not getting indexed?
What kind of search are you running?
Are there errors in the splunkd log?

0 Karma

pfcnetwork
New Member

Unfortunately not. Any further suggestions?

0 Karma

lukejadamec
Super Champion

did it work?

0 Karma

pfcnetwork
New Member

Corrections made, thanks

0 Karma

lukejadamec
Super Champion

Actually, let me check the last line....

I believe it should read:

`[default]
host = MGTNMS100

[splunktcp://9997]
Connection_host = none`

0 Karma

pfcnetwork
New Member

Thanks lukejadamec

I now have the inputs.conf looking like this:

[default]
[splunktcp://9997] Connection_host = none
host = MGTNMS100

I will let you know how it goes

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...