Monitor hosts

[ammul440]

I would like to monitor 10 hosts on a Splunk server. is that possible? What are the steps to monitor clients or hosts on Splunk server? should i install Universal forwader on all clients? I am confused and looking for suggestions.

[gcusello]

Hi ammul440,
at first follow the basic Splunk training and read Splunk documentation to understand how Splunk works.

Anyway, it isn't so immediate:

• you have to check if the firewall routes between target servers and Splunk server are open on ports 9997 and 8089,
• you have to install the Universal Forwarder on all the target servers,
• you have to choose the Technical Add-Ons (TAs) to use to ingest the wanted data or to create a nwe one,
• you have to deploy to all the Universal Forwarders the defined TAs to say to UFs which logs they have to ingest,
• at this point you could search these logs on Splunk, but you have to know how search logs on Splunk.

These are some starting links:
https://docs.splunk.com/Documentation/SplunkCloud/7.2.7/Data/Getstartedwithgettingdatain
https://www.tutorialspoint.com/splunk/index.htm
https://www.youtube.com/watch?v=6lX4DOd1T-s
https://www.youtube.com/watch?v=DJ6tXTsjX_A
https://www.youtube.com/watch?v=ZlKPqjuM0wo
http://www.splunk.com/view/SP-CAAAH9U
https://www.splunk.com/en_us/training/free-courses/splunk-fundamentals-1.html
https://docs.splunk.com/Documentation/Splunk/7.3.2/SearchTutorial/WelcometotheSearchTutorial

Ciao.
Giuseppe

[ammul440]

Hi Giuseppe,

Yes, I have read the basic documentation and some of the videos. I installed splunk universal forwarder on one of the client and set the forwarding data using the command, also configured the inputs and outputs.conf

./splunk add monitor /var/log/syslog -sourcetype systemlog

unfortunately, the data is not routing to the splunk server. on the splunk web interface the messages as fllows.

"The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. Learn more."
"Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly. "

please suggest.

[gcusello]

Hi ammul440,
At first, di you enabled receiving on Splunkserver?
in [Settings -- Forwarding and Receiving -- Receive Data] choose the port you used on UF to send data (default 9997), probably you have to restart Splunk.

If this doesn't resolve, please share the following files on $SPLUNK_HOME/etc/system/local:

• ouputs.conf
• inputs.conf.

Then some questions:

• Did you tested connection between servers?
• Which user you used to install Splunk UF?

Bye.
Giuseppe

[ammul440]

Hi Giuseppe,

Thanks for your quick response. Yes, I have enabled 9997.

inputs.conf
[default]
[monitor:///var/log/syslog]
sourcetype = systemlog

outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = 192.168.1.12:9997

[tcpout-server://192.168.1.12:9997]

1) I tested the connection, connection works but i guess port 9997 is closed on the target server and also on recieving server.

2) As a root, i installed splunk UF

Thank you.

[gcusello]

Hi ammul440,
probably the problem is on the connection.

If you want to ingest syslogs, use the sourcetype=syslog, so you'll have some default settings on Splunk Server.

Bye.
Giuseppe

[ammul440]

I used the iptables to add the port.

opt/splunkforwarder/bin/splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Password:
Active forwards:
None
Configured but inactive forwards:
192.168.1.12:9997

where 192.168.1.12 is splunk server . I am not sure whats wrong

[gcusello]

Hi ammul440,
did you tested ports using telnet from the target to the Splunk server?

telnet 192.168.1.12 9997

If it's ok, run this search on the Splunk server

index=_internal host=<client_hostname>

if you have logs, it means that the connection is ok but there's an input problem, if you haven't there's still a connection problem.

Then, sorry for the stupid notations:

• receiving must be enabled on Splunk server,
• inputs.conf and outputs.conf, must be on target server
• on target, you have to restart Splunk after modifying conf files

Ciao.
Giuseppe

[ammul440]

Hi please tell me

[ammul440]

ok seems there is a telnet issue

Trying ..
telnet: Unable to connect to remote host: Connection refused

when i ran this index=_internal host= i dindot get any output just the command worked..

I have enabled receiving on splunk server

what do you mean by target server (splunk server)?

Aslo, I have a doubt. Do we need to configure Indexer as a separate node?

[gcusello]

Hi ammul440,
the error you report says that you have to open firewall routes between targets and Splunk server on port 9997 for data and on 8089 for management (if you use Splunk server also as a Deployment server.
This is always the first check to do before to install a Universal Forwarder.

About the search I suggested, it has the objective to see if the target is connected to the Indexer.

I don't understand when you speak of a "separate node", target and Splunk server are on the same server?

Ciao.
Giuseppe