Getting Data In

Is it okay to run a universal forwarder without an inputs.conf?

xavierashe
Contributor

I am the security guy and Splunk admin. I am running 6.6.x universal forwarders on all my windows servers. I just found out that the server admins are cloning boxes all willy-nilly. When trying to figure out why SERVER05 wasn't reporting in, it was because its inputs.conf had "host = SERVER01". I was getting my data, it was just hiding under the wrong host.

Googling around I found the solution is to delete inputs.conf and server.conf, then restart the UF. This seems to work. The UF does recreate the server.conf, but not the inputs.conf.

My question is, is this a problem? All of my inputs are managed in apps via a deployment server. Do I need an inputs.conf that specifies the hostname? I can't see any problems right now but wanted to ask the community.

0 Karma
1 Solution

xavierashe
Contributor

I wanted to come back to this post to let everyone know that my solution works fine except for Stream. I use Stream to collect DNS logs. When i deleted inputs.conf and server.conf, the host came in as literal string"$decideatstartup". All other event types works as advertised.

Here's the script I wrote to clean up our environment. I have it scheduled to run once a week.

$Date    = Get-Date -Format yyyyMMdd
$logFile = "c:\work\logs\cleanupSplunk-"+$Date+".log"

function logger{
    param ([string]$element, [string]$message)
    $string = "$(Get-Date); $($element); $($message)"
    Add-Content -path $logFile $string
}

function fix-Splunk{
    param( [string]$server )
    $clean = $true
    if(![System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\bin\splunkd.exe")){
        logger $server "WARNING Splunk not found"
    } else {
        $splunkService = Get-Service -Computer $server -Name splunkforwarder
        if([System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf")){
            $clean = $false
            logger $server "INFO inputs.conf found"
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR deleting inputs.conf"
            } else {
                logger $server "INFO inputs.conf deleted"
            }
            Clear-Variable ProcessError
            Restart-Service -InputObject $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR restarting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder restarted"
            }
            Clear-Variable ProcessError
        }
        if([System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\deploymentclient.conf")){
            $clean = $false
            logger $server "deplymentclient found"
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\deploymentclient.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR deleting deplymentclient.conf"
            } else {
                logger $server "INFO deplymentclient.conf deleted"
            }
            Clear-Variable ProcessError
            Restart-Service -InputObject $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR restarting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder restarted"
            }
            Clear-Variable ProcessError
        }
        if ($splunkService.Status -eq "Stopped") {
            $clean = $false
            logger $server "WARNING Splunk service stopped"
            Start-Service $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR starting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder started"
            }
            Clear-Variable ProcessError
        }
        if ($clean -eq $true) {
            logger $server "INFO server clean"
        }
    }
}

logger "-" "INFO Script Started"
Get-ADComputer -ResultPageSize 10000 -ResultSetSize $null -Filter { 
    (OperatingSystem -Like '*Windows*Server*') -and
    (enabled -eq $True)
    } -ErrorAction SilentlyContinue | where { ($_.DistinguishedName -notlike "*OU=Domain Controllers,*") } | select-object -expandproperty name | foreach-object {
    Write-Warning "Trying $_"
    if (test-connection -computername $_ -count 1 -quiet -ErrorAction SilentlyContinue) {
        fix-Splunk $_
    } else {
        logger $_ "ERROR cannot connect"
    }
}
logger "-" "INFO Script Finished"

View solution in original post

xavierashe
Contributor

I wanted to come back to this post to let everyone know that my solution works fine except for Stream. I use Stream to collect DNS logs. When i deleted inputs.conf and server.conf, the host came in as literal string"$decideatstartup". All other event types works as advertised.

Here's the script I wrote to clean up our environment. I have it scheduled to run once a week.

$Date    = Get-Date -Format yyyyMMdd
$logFile = "c:\work\logs\cleanupSplunk-"+$Date+".log"

function logger{
    param ([string]$element, [string]$message)
    $string = "$(Get-Date); $($element); $($message)"
    Add-Content -path $logFile $string
}

function fix-Splunk{
    param( [string]$server )
    $clean = $true
    if(![System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\bin\splunkd.exe")){
        logger $server "WARNING Splunk not found"
    } else {
        $splunkService = Get-Service -Computer $server -Name splunkforwarder
        if([System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf")){
            $clean = $false
            logger $server "INFO inputs.conf found"
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR deleting inputs.conf"
            } else {
                logger $server "INFO inputs.conf deleted"
            }
            Clear-Variable ProcessError
            Restart-Service -InputObject $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR restarting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder restarted"
            }
            Clear-Variable ProcessError
        }
        if([System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\deploymentclient.conf")){
            $clean = $false
            logger $server "deplymentclient found"
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\deploymentclient.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR deleting deplymentclient.conf"
            } else {
                logger $server "INFO deplymentclient.conf deleted"
            }
            Clear-Variable ProcessError
            Restart-Service -InputObject $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR restarting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder restarted"
            }
            Clear-Variable ProcessError
        }
        if ($splunkService.Status -eq "Stopped") {
            $clean = $false
            logger $server "WARNING Splunk service stopped"
            Start-Service $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR starting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder started"
            }
            Clear-Variable ProcessError
        }
        if ($clean -eq $true) {
            logger $server "INFO server clean"
        }
    }
}

logger "-" "INFO Script Started"
Get-ADComputer -ResultPageSize 10000 -ResultSetSize $null -Filter { 
    (OperatingSystem -Like '*Windows*Server*') -and
    (enabled -eq $True)
    } -ErrorAction SilentlyContinue | where { ($_.DistinguishedName -notlike "*OU=Domain Controllers,*") } | select-object -expandproperty name | foreach-object {
    Write-Warning "Trying $_"
    if (test-connection -computername $_ -count 1 -quiet -ErrorAction SilentlyContinue) {
        fix-Splunk $_
    } else {
        logger $_ "ERROR cannot connect"
    }
}
logger "-" "INFO Script Finished"

ddrillic
Ultra Champion

-- Do I need an inputs.conf that specifies the hostname?

You don't need to explicitly specify the hostname - the forwarder runs uname -a on Linux and a similar command on Windows to find out, by itself, the host name.

s2_splunk
Splunk Employee
Splunk Employee

You should work with your server admins and have them run ./splunk clone-prep-clear-config if they clone an existing forwarder installation, then restart it (Documentation). That will not only initialize host settings, but also generate a new GUID for the installation.

0 Karma

xavierashe
Contributor

Yes, absolutely. That's the plan moving forward. But I've got fix a few hundred machines in the mean time.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You can easily create a new inputs.conf under /opt/splunk/etc/system/local

So essentially you will have multiple inputs.conf.. The one under etc/system/local just defines the hostname so you can identify it

Here's what it should look like

[default]
host = <HOST-NAME>

Restart splunkd after making this change

0 Karma

xavierashe
Contributor

True, but I've got to fix a few hundred machines. I need something scriptable. While I could do what you suggest, the real question is "do I need to."

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...