All Apps and Add-ons

PowerShell Modular input doesn't process my sourcetype for the data.

axl88
Communicator

I had a scripted input with power-shell as simply, *.bat files pointing to *.ps1 files and I was able to use my sourcetype by inputs.conf and props.conf.
We recently upgraded our system to Splunk 6.3 and decided to user PowerShell Modular input from UI since we would have a chance to edit schedules without restarting system etc..
Although, it is the same same power-shell script and props.conf, Splunk indexer fails to set sourcetype to my predefined type although I pick my source-type from the list. Instead, each time I save from UI as "from the list", whenever I go back to modular input page, it says manual and simply breaking my event into line by line.
Please advise as this is a bug for Splunk PowerShell modular input or I am missing something in the process?
Thanks upfront for your time.

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

According to the documentation you should be formatting your output differently when using in conjunction with modular input and powershell app.

http://docs.splunk.com/Documentation/AddOns/latest/MSPowerShell/Writescriptsforthemodularinput#Outpu...

Important: The modular input currently requires that any PowerShell scripts it executes produce output objects that do not have any script properties. Pipe output through | Select-Object * to ensure proper formatting.

 $Directories = @("c:\test","c:\windows")
 $now = $(get-date).ToString()

 foreach ($item in $Directories){
         $directoryInfo = $(Get-ChildItem $item -ErrorAction silentlycontinue| Measure-Object).Count
         $howManyDirectories = $(Get-ChildItem $item -ErrorAction silentlycontinue | where {$_.PSIsContainer} | Measure-Object).Count
         $howManyFiles = $(Get-ChildItem c:\ -ErrorAction silentlycontinue | where {$_.mode -notlike "d*"} | Measure-Object).Count
         if(!(Test-Path -Path $item )){
             write-output( $now + " no directory " + $item) | Select-Object *

         }
         elseif($directoryInfo -eq $directoryInfo){
             write-output($now + " " +$item + $onlyDirectory) | Select-Object *
             #I really dont understand this elseif and it's purpose  I have dir with one sub dir and it always gets stuck here.
         }
         else {
             $date = (Get-Date).AddHours(-4)
             write-output $((get-childitem $item | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| foreach-object { Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime | Select-Object *}))

         }
 }

You'll note I scratched your echos and same error handling variables you had and I dont understand the elseif. I think its meant to be if the directory is empty. I made you variables in your foreach equal to numbers instead of arrays with numbers attached which simplifies their usage later. Biggest point to get across here is the usage of the |select-object *

View solution in original post

jkat54
SplunkTrust
SplunkTrust

According to the documentation you should be formatting your output differently when using in conjunction with modular input and powershell app.

http://docs.splunk.com/Documentation/AddOns/latest/MSPowerShell/Writescriptsforthemodularinput#Outpu...

Important: The modular input currently requires that any PowerShell scripts it executes produce output objects that do not have any script properties. Pipe output through | Select-Object * to ensure proper formatting.

 $Directories = @("c:\test","c:\windows")
 $now = $(get-date).ToString()

 foreach ($item in $Directories){
         $directoryInfo = $(Get-ChildItem $item -ErrorAction silentlycontinue| Measure-Object).Count
         $howManyDirectories = $(Get-ChildItem $item -ErrorAction silentlycontinue | where {$_.PSIsContainer} | Measure-Object).Count
         $howManyFiles = $(Get-ChildItem c:\ -ErrorAction silentlycontinue | where {$_.mode -notlike "d*"} | Measure-Object).Count
         if(!(Test-Path -Path $item )){
             write-output( $now + " no directory " + $item) | Select-Object *

         }
         elseif($directoryInfo -eq $directoryInfo){
             write-output($now + " " +$item + $onlyDirectory) | Select-Object *
             #I really dont understand this elseif and it's purpose  I have dir with one sub dir and it always gets stuck here.
         }
         else {
             $date = (Get-Date).AddHours(-4)
             write-output $((get-childitem $item | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| foreach-object { Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime | Select-Object *}))

         }
 }

You'll note I scratched your echos and same error handling variables you had and I dont understand the elseif. I think its meant to be if the directory is empty. I made you variables in your foreach equal to numbers instead of arrays with numbers attached which simplifies their usage later. Biggest point to get across here is the usage of the |select-object *

axl88
Communicator

in Response to jkat54:
Assuming I have index "X" and sourcetype "Y", following code is in for each loop in the script:

$date = (Get-Date).AddHours(-4)
echo((get-childitem $item  *.* | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| %{ Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime}))

how do i suppose to override sourcetype or index? Why I can't set this up from Splunk side.
Script works perfect with .bat and .ps1 combination. It picks up the sourcetype i desire on Splunk.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Hello, some things have changed with relation to powershell scripts in version 6.3. The batch files are no longer needed, and there are some additional configuration items. Please see this link for more details:

http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/MonitorWindowsDatawithPowerShellscripts

0 Karma

jkat54
SplunkTrust
SplunkTrust

Example of using write-host alias (echo) vs using write-host:

alt text

0 Karma

axl88
Communicator

thanks for the answer. I checked the link and couldn't really identify anything particularly good for my question except overriding Splunk properties in powershell. I added code sample from my PS script in my question. Could you take a look and tell me what would make it work? any addition to powershell code?

0 Karma

jkat54
SplunkTrust
SplunkTrust

The echo makes no sense to me. $item isnt defined, $now isnt defined, string variable concatenation didnt look right.

How about this:

 $date = (Get-Date).AddHours(-4)
 get-childitem $item  *.* | where-object {$_.LastWriteTime -lt $now -and !$_.PSIsContainer}| foreach-object { $msg= '$(get-date) + " File: " + $_.FullName + "  LastWriteTime:   " +  $_.LastWriteTime'; write-output $msg}

may need write-host instead... i dont know

0 Karma

axl88
Communicator

I feel more like something buggy on Splunk side. Same script works perfectly with .bat file but not with Modular input :S

0 Karma

jkat54
SplunkTrust
SplunkTrust

Show me your batch script too please. And the full powershell.

I'll explain why one works and other doesn't.

0 Karma

axl88
Communicator

Man with batch script, ps1 works perfectly. Problem is ps1 only modular input doesn't pick the sourcetype. Here is full foreach loop:

foreach ($item in $Directories){
        #echo ( $now + " " +$item)
        $directoryInfo = Get-ChildItem $item -ErrorAction silentlycontinue| Measure-Object
        $howManyDirectories = Get-ChildItem $item -ErrorAction silentlycontinue | where {$_.PSIsContainer} | Measure-Object
        $howManyFiles = Get-ChildItem $item -ErrorAction silentlycontinue | where { $_.PSIsLeaf} | Measure-Object
        #dir $item -ErrorAction silentlycontinue
        if(!(Test-Path -Path $item )){
            echo ( $now + " " +$noDirectory + $item)

        }
        elseif($directoryInfo.count -eq 0){
            echo($now + " " +$noFileFound + $item)
        }elseif($howManyDirectories.Count -eq $directoryInfo.Count){
            echo( $now + " " +$item + $onlyDirectory)
        }
        else{
            $date = (Get-Date).AddHours(-4)
            #echo($now + " Directory:  " +
            #$item +
            echo((get-childitem $item  *.* | where-object {$_.LastWriteTime -lt $date -and !$_.PSIsContainer}| %{ Write-Output $now " File: " $_.FullName " LastWriteTime: " $_.LastWriteTime}))
            #echo dir

        }
}
0 Karma

jkat54
SplunkTrust
SplunkTrust

I believe the way you execute as batch is affecting the way you write your powershell. And so when you run your powershell using the cmd prompt and the batch file... it produces appropriate output, whereas when you run just the powershell from powershell, its not producing the expected results.

Try running it both ways and comparing the outputs.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If you find this to be true, then show me your batch and your powershell and I'll tell you how to re-write your powershell script so that it can stand on its own without command line interpreter overhead.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...