Recomendations on how to monitor Splunk systems

[borja_luaces]

Hello all,

This question might have been already addressed but here it I would like to know which one is the best approach for monitoring the systems where Splunk has been deployed in an distributed environment?

What i would like to monitor for example, is who is accessing the systems via SSH or what files are being modified,...

What I am thinking right now is installing a forwarder on each of the systems, point them to the deployment server and deploy the configuration from it but I was wondering if Splunk already does this by default or there is another better way to monitor this systems.

Regards,

[gcusello]

Hi @orja_luaces,
if you're speaking of monitoring of servers, you have to install on them a Universal Forwarder, a Splunk agent that permits to ingest logs from text fiels, scripts, and so on.
Which logs must be ingested is described in a configuration file called inputs.conf that's usually contained in a dedicated App called Technology Add-on.
To have a documentation about this read https://docs.splunk.com/Documentation/Splunk/8.0.1/Data/Getstartedwithgettingdatain

There are available hundreds of TAs just ready to be deployed, otherwise, you can create your own custom TA.
TAs are deployed to Universal Forwarders using a dedicated role/server called Deployment Server.
To have more infos about Deployment Serrvers, see at https://docs.splunk.com/Documentation/Splunk/8.0.1/Updating/Aboutdeploymentserver .

But in this way you solved only half of the problem, now you have your logs in Splunk Enterprise but you hae to find the security or error patterns (e.g. error messages) using the Splunk search features.
To have more infos about this, see the Splunk Search Tutorial ( https://docs.splunk.com/Documentation/Splunk/8.0.1/SearchTutorial/WelcometotheSearchTutorial ).

In addition, in community, you can find useful helps to your problems.

Ciao.
Giuseppe

[borja_luaces]

Thanks for the update Gcusello.

Thats what i want and this is why i am asking if i need to install a forwarder in the server or just modifying the inputs/outputs files of the current splunk installation will do the trick.

Regards,

[gcusello]

Hi @orja_luaces,
you don't need to install a Universal Forwarder if you already have a Splunk Enterprise instance.
It's always a best practice to send all the logs of all Splunk servers to Indexers, so you already should have set your Splunk instances to forward logs.
So you can configure inputs on your servers and take Linux logs from all of them.

If you need only /var/log/messages, you can enable the relative input by GUI on by one, otherwise, you could also install the TA_nix app to monitor all the server functions (cpu, memory, etc...) and deploy it manually or using the Deployment server or Master Node (if you have an Indexer Cluster) or Deployer (if you have a Search Head Cluster) .

Ciao.
Giuseppe

[borja_luaces]

I have been checking the _internal index but see no SSH connections to the Splunk server.

I do believe that the monitoring console does not provide the information i am looking for.

[gcusello]

Hi @orja_luaces,
In _internal index there are Splunk logs, not Linux logs
To monitor SSH connections, you have to monitor linux accesses in /var/log/messages ingesting these logs on each Splunk server and sending them to Indexers.
If instead you want to have web accesses to Splunk you can search in _audit index.

Ciao.
Giuseppe

[borja_luaces]

Thanks @gcusello.

I guess I did not explain myself, I am looking for monitoring the Splunk servers themself (search head, deployment server, indexers,...) not other servers.

I do not know if Splunk monitors the system it has been installed by default. I do know that it does monitor itself, this is the instance itself (indexers, ...).

Hope this clarifies a bit more the question.

[gcusello]

Hi @orja_luaces,
to monitor Splunk infrastructure use the Splunk Monitoring Console ( https://docs.splunk.com/Documentation/Splunk/8.0.1/DMC/DMCoverview ) and you already have all you need.

Ciao.
Giuseppe