All Apps and Add-ons

Splunk Add-on for Microsoft Powershell: How to troubleshoot why my Powershell script is not working on a heavy forwarder?

reswob4
Builder

I have a somewhat complex process I'm trying to get working. The synopsis is this: I have a report that generates a list of machines Splunk has not heard from in at least 12 hours. This report runs on the Search Head, a linux server. That report is piped to a CSV file using outputcsv. I then have a Heavy Forwarder running on a Windows server. On the HF, I wrote a powershell script that retrieves the CSV file, parses the machines, does some powershell 'magic', and then uploads the results to the SH in a new CSV as a lookup table in the Search app. The initial report works fine, and the Powershell script works when I run it from the command line of the HF. However, I want to automate the powershell script and I've been trying to do it in Splunk on the HF using the powershell add-on. I'm currently on 6.2 on all my servers.

So the add-on is installed on the HF and I created an inputs.conf file with the following:

[powershell://check-service]
script = . "c:\Tools\Powershell\test\check_service.ps1"
schedule = 30 */12 * * *
sourcetype = CheckService

Splunk is running on the Windows HF with the same account I'm doing the troubleshooting with. The execution policy for the account is unrestricted.

So I've looked through several of the entries talking about troubleshooting powershell scripts and so my first question is this: I wanted to look at the errors and one of the other entries said to check the powershell logs by running the following search:

index=_internal source="*powershell*.log"

But running this on the SH yields no results. Am I searching for the powershell logs in the wrong place? Are there no logs because I haven't set up the add-on correctly?

I also tried index=_internal source="*powershell*" and index=_internal source="*.ps1" but neither of those searches yielded anything.

Any other suggestions on how to troubleshoot? Any ideas on why this script isn't running?

Thanks.

0 Karma
1 Solution

javiergn
SplunkTrust
SplunkTrust

Have you tried running those searches locally on the heavy forwarder?
Simply enable the GUI and give that a go.

If that search works fine then you might need to enable the _internal log forwarding to your indexers in order to have it searchable from your Search Heads. Take a look at this answer.

In any case, I think your problem is with the script line. Take a look at the following block from the inputs.conf documentation. I think Splunk expects the script to be in a different directory:

[script://<cmd>]
* Runs <cmd> at a configured interval (see below) and indexes the output.  
* The <cmd> must reside in one of:
  * $SPLUNK_HOME/etc/system/bin/
  * $SPLUNK_HOME/etc/apps/$YOUR_APP/bin/
  * $SPLUNK_HOME/bin/scripts/
* Script path can be an absolute path, make use of an environment variable such
  as $SPLUNK_HOME, or use the special pattern of an initial '.' as the first
  directory to indicate a location inside the current app.   Note that the '.'
  must be followed by a platform-specific directory separator.
  * For example, on UNIX:
        [script://./bin/my_script.sh]
    Or on Windows:
        [script://.\bin\my_program.exe]
    This '.' pattern is strongly recommended for app developers, and necessary
    for operation in search head pooling environments.

View solution in original post

javiergn
SplunkTrust
SplunkTrust

Have you tried running those searches locally on the heavy forwarder?
Simply enable the GUI and give that a go.

If that search works fine then you might need to enable the _internal log forwarding to your indexers in order to have it searchable from your Search Heads. Take a look at this answer.

In any case, I think your problem is with the script line. Take a look at the following block from the inputs.conf documentation. I think Splunk expects the script to be in a different directory:

[script://<cmd>]
* Runs <cmd> at a configured interval (see below) and indexes the output.  
* The <cmd> must reside in one of:
  * $SPLUNK_HOME/etc/system/bin/
  * $SPLUNK_HOME/etc/apps/$YOUR_APP/bin/
  * $SPLUNK_HOME/bin/scripts/
* Script path can be an absolute path, make use of an environment variable such
  as $SPLUNK_HOME, or use the special pattern of an initial '.' as the first
  directory to indicate a location inside the current app.   Note that the '.'
  must be followed by a platform-specific directory separator.
  * For example, on UNIX:
        [script://./bin/my_script.sh]
    Or on Windows:
        [script://.\bin\my_program.exe]
    This '.' pattern is strongly recommended for app developers, and necessary
    for operation in search head pooling environments.

reswob4
Builder

Sorry, I get it now. So I did run the _internal searches locally on the HF and I did not get any results. I used the link you sent and made the changes and now the internal logs are being forwarded correctly.

I then moved my ps script to $SplunkHome\etc\system\bin on the Windows HF and changed the script line in my inputs.conf to:

script = . "$SplunkHome\etc\system\bin\check_wls_beta2.ps1"

The script now runs, but albeit with some errors. The errors have more to do with creation and access to certain files than with Splunk.

I'm going to mark this as resolved. Thanks.

0 Karma

reswob4
Builder

I need to ask a basic question, how do I run the powershell script from the search?

I'm looking for an example in answers, but haven't found one yet....

Thanks.

(To reiterate, it does run from the command line of the windows server no problem)

0 Karma

javiergn
SplunkTrust
SplunkTrust

Sorry if my comment wasn't clear enough, what I meant by this:

Have you tried running those searches locally on the heavy forwarder?
Simply enable the GUI and give that a go.

Is that you should try to run your _internal searches from the heavy forwarder just to make sure you are not missing any configuration that forwards those logs to the indexer. By default your _internal logs might not be forwarded automatically to the indexers.

You can't run powershell from the search GUI.

In any case, did you try what I mentioned about the script path? I think you need to place your file somewhere within your SplunkHome directory to get this running. I don't think Splunk is going to run scripts that are outside these locations:

  • $SPLUNK_HOME/etc/system/bin/
  • $SPLUNK_HOME/etc/apps/$YOUR_APP/bin/
  • $SPLUNK_HOME/bin/scripts/
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...