Why is Splunk DB Connect 2 not rotating py_health.log on a Windows Heavy forwarder

[ragedsparrow]

Splunk Version: 6.3.3
OS Version: Windows Server 2008 R2 Enterprise 64-Bit
DB Connect Version: 2.1.3

It seems when the py_health.log file reaches 10,000KB, it is not rotating out and seems to lock. This is causing issues with DB Connect connecting to the databases as well as running the queries. Has anyone else had this problem? I can stop Splunk and roll it manually, however it isn't ideal. Has anyone else had this issue?

[ragedsparrow · Accepted solution]

So, it seems that since this is a Windows OS, the file, when in use can not be modified or deleted. My workaround was to create a scheduled task to run a rotate script that will stop Splunk, remove the backup, rename the file, and start Splunk. This is probably not the best way to do it, but it's the only way that I've found so far.

$splunkProcess = Get-Service | Where-Object {$_.Name -eq "Splunkd"} | Where-Object {$_.Status -eq "Running"}

if ($splunkProcess){

Write-Host "Splunk is running..beginning process"
$StopCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' stop
'@
Invoke-Expression -Command:$StopCommand

    if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
        Write-Host "Splunk stopped, deleting old log file"
        Remove-Item -Path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
    }
    else{Write-Host "No file backup found"}
    Write-Host "Renaming log file"
    Rename-Item 'C:\Program Files\Splunk\var\log\splunk\py_health.log' 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
$StartCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Write-Host "Starting Splunk"
Invoke-Expression -Command:$StartCommand

}

else {
if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
Write-Host "Splunk is not running"
$command = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Invoke-Expression -Command:$command

}

This was pretty much just banged together to hold things together for now. I will be adding in some exception handling as well as removing the Write-host and will be instead writing to a log file.

[jcoates_splunk]

This was just fixed in version 2.3.0.