Getting Data In

Shell Script to Check if Multiple Servers are Correctly Forwarding Data

mmensch
Path Finder

Hi all,

I have a list of servers in a text file "servers.txt."

I am trying to create a shell script that will see if all of my servers are sending data to Splunk on a daily basis. I have some scripting knowledge and created a script to see if the server is up or down by simply pinging it. I want to take this one step further. I want to know when the Splunk Forwarder has stopped working or something along those lines.

Any help would be appreciated.

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

I think it would be a lot easier to verify the completion of data being forwarded by searching 1.) the index the data is being forwarded to, and 2.) the _internal index to verify connection. Then, if you don't see data in 1, you can check 2 to make sure the connection is alive, and if 1 & 2 are blank, then you can investigate further.

Splunk is actually really great at doing this, so I would suggest avoiding a scripting language or shell scripts if possible.

0 Karma

mmensch
Path Finder

I understand Splunk is great at doing this, but this does not scale to thousands of servers. I have a list of 250 servers currently and it will grow into the tens of thousands.

0 Karma

lycollicott
Motivator

You could script a remote "splunk status"

$serverlist = get-content splunk80uf.list

foreach ($server in $serverlist)
{
$result = Invoke-WmiMethod -ComputerName $server -Class Win32_Process -Name Create -ArgumentList "cmd.exe /c e:\app\SplunkUniversalForwarder\bin\splunk.exe status >E:\app\status80uf.log" 
if ($result.ReturnValue -ne 0)
    {
       ##$exception = New-Object System.ComponentModel.Win32Exception([int]$result.ReturnValue)
        ##Write-Error "Error launching installer on computer ${server}: $($exception.Message)"
        echo Crap
    }
}

sleep 5

foreach ($server in $serverlist)
{
write-host "$server`t" -nonewline
Get-content \\$server\e$\app\status80uf.log
}
0 Karma

lycollicott
Motivator

On Unix/Linux you could take a similar approach if you setup ssh keys.

ssh user1@server1 command1

0 Karma

mmensch
Path Finder

This is not working for me.

0 Karma

lycollicott
Motivator

What kind of errors or messages do you get?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...