Getting Data In

Universal Forwarder not displaying data on SplunkWeb on another server

atewari
Path Finder

We have two Linux servers using Splunk 5.0.1 on 64-bit.

  1. A full Splunk install (SplunkD and SplunkWeb). We created a Receive data port 8001 on this.
  2. Universal Forwarder on second Linus server. We added a forwarder using command
    
    splunk add forward-server server1:8001 -auth admin:somepassword
    
    

It was successfully added. We restarted the forwarder on server2. So we ran the command



splunk list forward-server

This command showed that server1:8001 was added but was not active. When we ran the list command again it said the file was locked. The metrics.log file says it is connected successfully.

But how do we view server2 data on SplunkWeb running on server1? We added the *nix App, but it cannot see server2 selection anywhere. We only see server1 info. Is there another step to activate the forwarder on server2 and enable something on server1 to view server2 logs?

The "deploy" forwarder documentation is confusing. It gives a few commands and then asks to test the deployment of the forwarder without instructions on what to test.

Can anyone point us to the next steps - links, answers, anything?

Thanks

Tags (1)
0 Karma

atewari
Path Finder

As it turns out, internally, 8089 and 8001 were not communicating. The log files did not indicate that. After reviewing the firewall (iptables.rules.up), again and again, I just re-wrote the firewall rules again. Carefully reviewing the syslogs indicated that there was something being rejected by 127.0.0.1:8001. Now it all works. Thanks Drainy and Dave for your responses and support.

0 Karma

DaveSavage
Builder

Ah - layer 2, gets me every time! 😉 Glad its sorted out and you are welcome. Drainy did the leg work, mine were only a few prompts re *-nix. Good luck!

0 Karma

DaveSavage
Builder

Drainy and Atewari - I stuck my hand up earlier on a feature of star-nix but not sure if you clocked it, or didn't need it...no worries either way. You said 'We added the star-nix App", but it cannot see server2 selection anywhere'....well, tbh you won't until you get the index 'os' sorted out. Traffic to *nix is expected in that space.
Cheers, Dave

0 Karma

DaveSavage
Builder

Pls confirm the above. It will happen, trust me. There are some great resources here plus the whole of Splunk.
Br, Dave

0 Karma

DaveSavage
Builder

The forwarder is up, running, and sending logs...?
There are no firewall issues i.e. you can definitely see traffic to the indexer? Are they on the same sub-net?
You plan to use *-nix in the future which needs data in the os index - but let's resolve your connectivity issues first.

Where do we go now?
Is there ANYTHING being indexed on the Splunk search home page? What sources, hosts? Is it just local stuff e.g. the indexer server?
The ports being used (8001 above) are definitely configured as send to on the forwarder, and in a TCP stream in the inputs section of 'Manager' on the indexer?

0 Karma

DaveSavage
Builder

what we know is that the data is being forwarded (you are sure about this atewari)? I.e. you can see it....where?
It is uncertain whether it is being indexed, or within which index?
You have applied all of the inputs and outputs suggested.
We don't know what data or logs you expect to have been sent from #2, but Drainy & I are probably assuming that its just standard Linux logs...i.e. the logs are being generated and are readily available?

0 Karma

DaveSavage
Builder

This one just refuses to roll over and get solved, doesn't it?! Usually @atarewari it IS very easy to be honest - at least iro getting data in...and prima facie your spec is simple. We all have complications but usually of our own derivation / environment on the non-standards - but thats later. Stick with it - Splunk is an awesome tool.
So to recap -

0 Karma

Drainy
Champion

Yeah, since atewari's inputs didn't contain anything to do with the *nix app nor did they mention it I skipped over it, a valid point though as it would show the same symptoms.

0 Karma

atewari
Path Finder

Thanks Dave. At this point I am not even looking at *-nix. I am looking at Spunk's summary page. I added the index = os in the inputs.conf file and restarted both forwarder and splunkWeb. It did not do anything. Reviewing the log files, I know data is being transferred from server2 (univ. forwarder) to server1 (fullinstall).

I followed all instructions in the inputs.conf spec with no luck.

Splunk has great features, and we were hoping for an easier configuration. Any last suggestions that you think we can try?

thanks again for your help! We really appreciate it.

0 Karma

DaveSavage
Builder

Your 2nd server with the forwarder on it needs to be firing at index 'os'...that's all. If its only those 2 servers then 9997 is fine, but we segregated out star-nix traffic from Windows, sending the former to (say) port 9996, then told the indexer's Data Inputs params (see Manager if you prefer GUI) under 'More Settings' to populate the index 'os'. We work in a highly secure environment complying to auditable Government levels, so need both.

0 Karma

DaveSavage
Builder

Atewari - not wishing to tread on Drainy's toes 'cos he is a good 'un and very solid....my two-pennneth relates purely to that of experience with the star-nix (am still trying to work out the tagging variables here) plug-in. Level playing field - you say summary page - by that you mean star-nix's...or Splunk's Search page? Portal being standard Splunk access?? We don't tend to use that word too much around here, albeit it will be a portal for someone, being web access.

0 Karma

atewari
Path Finder

Dave,
Do you see any issues with conf file changes we made for Splunk not to display both servers on summary page? I have increased the debugging to determine what is causing the forwarder server host not showing up on summary page of the portal.

Any insight would be greatly appreciated, thanks

0 Karma

atewari
Path Finder

BTW, index=summary did not show any data.

0 Karma

atewari
Path Finder

Thanks! We have confirmed that the forwarder has established connection and is sending data to the fullinstall server1. As Drainy pointed out, we may have left some configuration out. But we are not sure which stanza is missing to display the the universal forwarder host on the summary page.

Any suggestions? We have posted our conf files below.

0 Karma

Drainy
Champion

I don't think you need that tcpout-server, but thats beside the point. Why are you sending it all into index=summary? delete that and let it drop into the default main index, the search app summary page won't show anything from indexes other than main.

As another test, try doing a search for index=summary and see if your data appears.

0 Karma

Drainy
Champion

D'oh. You aren't monitoring any files on the forwarder.... so it hasn't got anything to send, unless you're holding out on me here? 🙂 You need to add a monitor statement to your inputs.conf on the forwarder for it to actually monitor anything.

0 Karma

atewari
Path Finder

BTW, index=summary did not show any data. I also changed the tcp to splunktcp

[default]
host = fullinstall.xyz.com
disabled = 0


[splunktcp://uf.xyz.com:8001] disabled = 0

I get the following info in splunkd.log on the forwarder server

11-30-2012 09:26:16.412 -0600 INFO  loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
11-30-2012 09:26:16.504 -0600 INFO  TailingProcessor - TailWatcher initializing...
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
11-30-2012 09:26:16.505 -0600 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
11-30-2012 09:26:16.525 -0600 INFO  TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:8001
11-30-2012 09:26:46.050 -0600 INFO  CMConfig - A splunktcp forwarder port is not configured in inputs.conf

In any case should I not see raw data if I used tcp instead of SplunkTCP? What does the splunktcp forwarder port not configured mean? Are there missing stanzas in the input.conf of forwrader server or fullinstall server?

0 Karma

Drainy
Champion

Well, did you try the search for index=summary? Once your data has been forwarded/indexed it won't send them again unless you do a few cleanup tasks. Also that should be splunktcp:// instead of tcp:// in your inputs on the indexer.

0 Karma

atewari
Path Finder

Drainy,
Thanks for your quick response. Here is what we now have in the inputs.conf on fulliinstall.xyz.com



[default]
host = fullinstall.xyz.com
disabled = 0

[tcp://uf.xyz.com:8001]
disabled = 0


No difference. could it be that all logs are sent to the same server and therefore two hosts are not shown.

We changed the inputs.conf [default] stanza to

host = anotherserver.xyz.com

When we did this, it showed this new host. But it collects data for the fullinstall.xyz.com

Are we missing anything else?

0 Karma

atewari
Path Finder

BTW, index=summary did not show any data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...