Newbie getting set up (monitoring several computers-long post)

[Rikakiah]

I'm a fairly new admin and extremely new at looking at reports/data. I have an issue with my server that I can't track down and was hoping if I could tell everything that was going on with all the computers at the time I lose service, I might figure out the cause. It's an xserve running DNS, OD, AFP, supporting about 20-odd mac clients (all network accounts with home folders on the server). Basically, the server simply hangs (no crash or error report or anything in any logs I can find on the server, just no one can log in and it shows as unavailable from my ARD machine. Also, I can't log in using local admin credentials--have to restart from another machine using Server Monitor. It happens randomly during off hours and I'm having trouble isolating even what time anything wrong happens, let alone what may be causing it.

I have my ARD machine running Splunk (since it never goes down) and the server is forwarding. I want to forward all (or at least several) of my other computers as well, but the ARD machine never sees any of the others. They should all be forwarding to the same port on the receiving machine, right?

Finally, I could use some help getting something set up to see data that can narrow down my problem. I'm thinking something that shows DHCP activity, AFP connection info, and any other network requests directly to/from the clients to the server (or vis versa), but I'm not sure what commands I should be looking at...

Like I said, I'm very new to this, so any help would be appreciated.

EDIT: Can't seem to comment anywhere else to give info for some reason... I was trying to forward with Splunk forwarders (thought that was the only way?). I got a LightForwarder set up on the server and it appears to be working (I have data about my server showing up on ARD machine where the Splunk web app is running, although I'm not entirely sure yet how to filter out exactly what data I want to see). Not sure on your last question, gkanapathy--I assumed the LightForwarder sent everything...? But, since I'm getting at least some sort of info from the server, I'm assuming the receive port is open (I only need to use one receiving port for all the forwarders, right?).

Also not really understanding your questions, mayler. Are you asking how I'm trying to connect them for Splunk? Please explain exactly what you need so you can help me more in small words since I'm still quite new to the inner workings of connections. Where would I go to check/setup my input configuration?

[gkanapathy · Accepted solution]

It sounds to me like you're getting a little ahead of yourself trying to jump right into things, and it may benefit you to step back and make sure you understand the concepts and mechanisms for getting data into Splunk. I don't think this forum is a good format for broad conceptual overviews, so you may be better off going here http://docs.splunk.com/Documentation/Splunk/5.0/Data/WhatSplunkcanmonitor and here http://www.splunk.com/wiki/Community:Getting_data_into_Splunk and following through on network and file monitoring, and then how forwarding works.

You might then have more specific questions about this or how it works that can be addressed here, once there is a base of common understanding of the terminology and concepts.

[Rikakiah]

Yep, I admit I mainly need help with the basics right now. I browsed through some of that (although some of that doesn't look familiar--must've overlooked that page). I've also been rushed with a lot of other projects going on right now and just wanted it up and running. I'll have more time next week to really focus more on this specifically. Thanks.

[BunnyHop]

You can go to the machines where the LightWeightForwarder are installed and run the command ./splunk list forward-server. If the forward-server is listed as inactive, most likely, the server does not have the port open to receive the Splunk tcp traffic.

[gkanapathy]

Can you tell us how specifically you are forwarding? e.g., Splunk forwarders, syslog? And what data and how are you collecting it?

[mayler]

Are your other boxes using syslog to send to your ARD machine? How is your input configured? Port or File/Dir? If port, port 514?