Not getting any Data with Xenapp.

[joefixit71]

I installed a Instance of Splunk, setup 3 servers with the forwarders installed pointing to the main instance on port 9997. (License server, XenApp Data Collector/Xml broker and regular Xenapp server. I copied the folders for the snapins for each type of server to the "C:\Program Files\SplunkUniversalForwarder\etc\deployment-apps" The Index is installed for xenapp since it gets installed during the xenapp snapin install but its not receiving any data.
I get the message "no matching fields exist" on top and "No results found" I don't see any Farms listed to click on so its not even connecting to the Farm servers I tried to add. I don't have any firewalls in between so its not blocking any ports. I setup a receiver listening on port 9997 on the Main instance but still no data.
This below is the Output file of a server thats forwarding data.
[tcpout]
defaultGroup = default-autolb-group

[tcpout-server://nwnifictx040.usa-ed.net:9997]

[tcpout:default-autolb-group]
disabled = false
server = nwnifictx040.usa-ed.net:9997,nwnifictx040:9997

[tcpout-server://nwnifictx040:9997]

This is a server forwarding data's Input file
[default]
host = NWNIFICTX030

let me know if any other information is needed.

This is a evaluation setup that I was really trying to get a good look at before tomorrow.
thanks,
Matt

[bigtyma]

You might need to adjust firewall rules to allow the forwarders to reach the indexers over tcp/9997

[joefixit71]

Real Time doesn't show anything. Basically the Farm is showing up but not much else. Also I launched a few of the powershell scripts manually wondering if thats why I see some information now.

[joefixit71]

I uninstalled the Splunk instance and installed in on another server. I also uninstalled the forwarders and reinstalled and pointed them to the new splunk instance. I copied the snapins to C:\Program Files\SplunkUniversalForwarder\etc\apps on the xenapp,xml broker, and license server. I ran the Xenapp SDK script installs on all the Forwarders as well as the Main Splunk Server. The service on port 9997 seems to be staying up like its supposed to on the main instance server. I now see a Farm to choose from in Environment tab. However not all information is showing up.

[joefixit71]

no firewalls in between. Something might be with the service not staying up and listening

[bigtyma]

Check your PowerShell execution policy, I believe you want 'RemoteSigned'

[jconger]

The Splunk Windows service needs to run as a XenApp farm administrator.

Are you able to run the PowerShell scrips manually?

Anything in "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log"?

Does the following search produce any results:
index=xenapp*

[jconger]

Perhaps something else is using the port or another configuration file has the listener disabled.

To test, I would modify the inputs.conf file to use port 9998. Then, update your forwarders to send their data on port 9998.

[joefixit71]

09-27-2013 15:55:20.789 -0400 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0.
09-27-2013 15:55:20.789 -0400 INFO HttpPubSubConnection - Could not obtain connection, will retry after 60 seconds.
09-27-2013 15:55:22.928 -0400 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected

some lines in the splunkd.log file on one of the forwarders

[joefixit71]

09-27-2013 15:55:13.010 -0400 WARN TcpOutputFd - Connect to 172.18.98.55:9997 failed. No connection could be made because the target machine actively refused it.
09-27-2013 15:55:13.010 -0400 ERROR TcpOutputFd - Connection to host=172.18.98.55:9997 failed
09-27-2013 15:55:13.010 -0400 WARN TcpOutputProc - Applying quarantine to ip=172.18.98.55 port=9997 _numberOfFailures=5

[joefixit71]

ok, seems my 9997/Receiver service isn't staying up as i came back later and nothing was listening on port 9997 and had to restart splunkd

but I do see two established connections on port 9997 one of the local Spunk instance server an a Xenapp server that has a forwarder setup..

[jconger]

What is in your splunkd.log on your forwarder now?

[joefixit71]

OK, I can telnet now so service is listening not really sure what changed there but I see 9997 listening since i have a receiver setup on that port.

change the one in this folder? C:\Program Files\Splunk\etc\apps\SplunkAppForXenApp\local
it already had the first two lines. I did change this one but same results..

Still see No Matching Fields exist and NO results found. Shouldn't I see a list of FARMS in the list of FARMS that I can click on in the Environment web page once configured properly?

thank you!

[jconger]

Gotcha.

There isn't a problem running Splunk on a XenApp server.

Does running netstat -a on your Splunk server show that it is listening on port 9997?

Make sure the following is in your inputs.conf file on your Splunk server:

[splunktcp://9997]
connection_host = ip
disabled = 0

[joefixit71]

Yes, this Splunk Instance server is a server that does have Citrix Xenapp installed and the server is our admin server, which also has edgesight/Desktop Director for DEV. There are three other servers; a Citrix Xenapp server,an XML broker server, and a License server which are different boxes than the Instance server that have the Forwarders installed. I was just trying to make a point that the 9997 wasn't allowing connections from the Splunk server it was listening on so can't be a firewall.

from saying that are you saying that Splunk will not work on a server that has XenApp installed?

[jconger]

Splunk should be installed on a separate server from your XenApp server. The Splunk Universal Forwarder goes on your XenApp server.

Therefore, you should be able to telnet from your XenApp server to your Splunk server on port 9997.

[joefixit71]

Can't even telnet locally on the Splunk INnstance server to the 9997, like I can to the ICA 1494 port. So "telnet localhost 9997" doesn't work, but "telnet localhost 1494" does work.

[joefixit71]

Thats what I can't understand, when i first saw the message in the log I thought maybe that a firewall somehow got enabled locally (which isn't standard in our environment) or something was blocking since I couldn't telnet to port 9997, but there isn't a host based firewall on any of these servers.

[jconger]

The error in splunkd.log indicates the indexer either has a firewall blocking the incoming connection or that the port is not opened.

You indicated that you set up receiving on port 9997. Does the OS that the indexer is running on have a firewall enabled?

Can you telnet from your XenApp server to the Splunk indexer on port 9997?

[joefixit71]

The Splunk windows services is setup as Administrator to the farm.
I can run Powershell scripts on the Farm

Index-"xenapp" produced 0 matching events.

Here is a line from the splunkd log that seems to be the problem but not sure what to do to fix..

172.18.98.55:9997 failed. No connection could be made because the target machine actively refused it.

thanks..

[jconger]

Did you set your PowerShell Execution Policy to "Remote Signed"?