All Apps and Add-ons

1 of 12 universal forwarders not getting through

Techfrogger
Explorer

I've set up universal forwarders on 12 identical windows XP boxes and all are using the default port. Back on Splunk, using the Windows app, I can only see 11 of the 12 and I've verified that all settings are identical for the forwarder. Any idea where to look, or what to look at? Splunk server is installed on a Ubuntu box, version 12.04.1 LTS

0 Karma

MarioM
Motivator

what do you have in your culprit forwarder splunkd.log?

0 Karma

Techfrogger
Explorer

Still no answers, but now I have more news: this whole time I've been looking at Splunk via the Windows app. But if I ignore that and just search on the workstation name, thousands of items show up. So now it seems that Splunk IS getting the info I want but for some reason the Windows app for Splunk refuses to display info for this one workstation. How strange is that?!! Can anyone troubleshoot the windows app?

0 Karma

Techfrogger
Explorer

As to the question: what do you have in your culprit forwarder splunkd.log? I don't know the location of that log. Could you point me in the right direction?

0 Karma

Techfrogger
Explorer

Yes, that's correct: all 12 machines have identical configurations and are on the same subnet. And all data, from all hosts are sending to same index. Same configurations deployed to all agents. Really baffling.

0 Karma

lmyrefelt
Builder

All data, from all hosts coming to same index ? Same configuration/s deployed to all agents?

0 Karma

dart
Splunk Employee
Splunk Employee

And do you see those same sourcetypes for all 12 hosts?

0 Karma

Techfrogger
Explorer

WinEventLog: security, App and System

0 Karma

dart
Splunk Employee
Splunk Employee

What is the source and sourcetype for the data you are receiving?

0 Karma

Techfrogger
Explorer

I just ran netstat and it does show a connection to all 12 hosts. I then took a look at the firewall, ufw, and it shows activity from all 12 too. This is really baffling. I just installed some updates to the box and then rebooted, hoping that would be the end of it, but once again, all show up except the one in question.

0 Karma

dart
Splunk Employee
Splunk Employee

If you run netstat on the ubuntu box, does it show connections from all 12 hosts?

0 Karma

Techfrogger
Explorer

Sadly that's not the case. All are on the same subnet and the firewall they go through shows that all 12 are sending data. I assume the problem is on the Ubuntu server but that's where my understanding stops. Anyone know how to troubleshoot Ubuntu? Thanks everyone

0 Karma

ARothman
Path Finder

I'm unsure of what troubleshooting you have already tried, but I would start at the network. While understanding that you have 12 identical WindowsXP boxes, all using the default port, are they all on the same subnet with the same network rules applied to them? It sounds to me like this one you are having problems with may be behind a firewall that is blocking the forwarder traffic.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...