Getting Data In

Scom integration to Splunk

officeguy
Explorer

Hi,

What is the actual process by which the events are 'forwarded' into splunk. The ps1 script gets events and does a write-host of the alert data to the console... but... how does splunk then 'notice' that these events are written to the console of the powershell session?

Tags (2)

MartyLindsay
New Member

Resolved:

in scom_client.ps1 replace the "p.m" format with "pm".

    $scomdate = ($event.TimeGenerated).ToLocalTime();
    $scomdate = $scomdate -replace("p.m","pm")
    $scomdate = $scomdate -replace("a.m","am")
    write-host $scomdate;

and

    $scomdate = ($alert.TimeRaised).ToLocalTime();
    $scomdate = $scomdate -replace("p.m","pm")
    $scomdate = $scomdate -replace("a.m","am")
    write-host $scomdate;

no doubt I could have fixed this within propf.conf also, but for now this works as expected.

0 Karma

officeguy
Explorer

another question.
I have made the powershell script populate the time as LocalTime by doing this:

$scomdate = ($event.TimeGenerated).ToLocalTime();
write-host $scomdate;

this is required as my local timezone is UTC+13

and the date arrives at splunk as text like this:
30-10-2012 3:10:29 p.m.

this works fine in the morning, however... once the dates go past 1pm, they get logged as if they were a.m, instead of p.m
i.e the text in the event is 30-10-2012 3:10:29 p.m. but the logged time for the event is 30/10/2012 03:10:29.000

my props.conf is like this:

[source::SCOM...]
SHOULD_LINEMERGE = false
TIME_FORMAT=%d-%m-%Y %I:%M:%S %p
MAX_TIMESTAMP_LOOKAHEAD=43

any ideas how I force splunk to correctly apply the a.m and p.m ?

0 Karma

officeguy
Explorer

I now have this working. My issue was one of credentials. My splunk install on windows was running splunkd as LocalSystem, but my own user account doing the testing was the one which had permission to access the SCOM server via powershell. To see the actual error I had to start a powershell session using AT, then manually run the powershell script. My workaround for now is I have changed the splunkd service to run under my own user account.

Also, one improvement to the code. in scom_client.ps1 make the following change:

#write-host ("ComputerName=" + $alert.MonitoringObjectDisplayName);
write-host ("ComputerName=" + $alert.NetbiosComputerName);

this is more accurate, as the monitoring object display name can contain things like "Terminal Services" and other such objects. Sadly it isn't a full fix, and some alert types are not related to hosts, such as Active Directory objects etc. I'll post a better solution shortly.

0 Karma

officeguy
Explorer

thanks for the info. I have now narrowed this down to a difference between how splunk reads file paths, compared to how powershell reads file paths. i.e parts of the powershell script can find the file when run manually, but when splunk runs the same script, it can't find the file. I'll keep testing variations on this, and post the final config changes once I have it working.

0 Karma

bmacias84
Champion

If that doesn't work edit scom_client.path with the windows equivalent for pathing. Final suggestion is create a bat file that mimics the .path file. modify input for path file to the bat file.
[script://.\bin\scom_client.bat]

Also check your Splunkd.log file there will be some clues in there if the script is failing at the inputs.conf.

0 Karma

bmacias84
Champion

Scripted inputs work by reading any thing outputted to the console by -->[script://.\bin\scom_client.path] into splunk.
My next guess is that your running this on a windows server. Some of the config pathing does always work the same in Windows. lower and uppercasing drive partition letter can cause some headache.So try editing the inputs.conf with absolute path c:\Program Files\Splunk\etc\apps\scom\bin\scom_client.path.

0 Karma

officeguy
Explorer

log output.. all seems well..
02-10-2012 1:22:39 p.m. INFO Alerts checkpoint loaded successfully
02-10-2012 1:22:39 p.m. DEBUG Get-Alert -criteria "TimeRaised > '10/01/2012 21:50:04' AND Severity = '2'"
02-10-2012 1:22:40 p.m. DEBUG Forwarding 125 alert(s)

however.. this mention of 'Forwarding' is only a log entry. the 'action' to forward it to splunk is only the write-host action within powershell.
My main question is therefore : what configuration within the app actually tells splunk to look for the output from the powershell command?

0 Karma

officeguy
Explorer

thanks. feedback below. 1:yes, 2:yes, 3:yes 4:this doesn't run, but most likely this is due to syntax which isn't supported via the command line in windows: $SystemRoot\ and $SPLUNK_HOME. I assume these should be environment variables like %SystemRoot%. I have change this to %SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe -command "& 'C:\Program Files\Splunk\etc\apps\scom\bin\scom_client.ps1'" which runs ok, gets data, updates the checkpoint file and the log file. 5:loggin already enabled... will put output into next comment..

0 Karma

bmacias84
Champion

Troubleshooting:
1-Make sure app in app manager is enabled.
2-make sure inputs.conf in the %splunk%\etc\apps\scom is enabled and interval is correct.
3-make sure scom_client_xml has the correct ip or fqdn and pollinterval matches input.conf.
4.run the scom_client.path manually see if you get any output. if not run the scom_client.ps1. still nothing does your scom server allow remote execute or RPC.
5. still having problems edit the scom_client.xml loggingEnabled value to '1' which will create a log in %splunk%\etc\apps\scom\bin.

Hope this helps

0 Karma

bmacias84
Champion

Ive just install the scom app in my test enviroment and this app has a few of moving parts.

inputs.conf-which runs the scom_client.path
scom_scom_client.path-refers to ps.exe and script.
scom_client.ps1-which refers to scom_client.xml

All of which you probably already know. inputs.conf defines a script input that runs on a 60 sec interval (or 60sec cron job) calling the scom_client.path. Ever 60sec scom_client.ps1 is called reading the inputs directly off the console. scom_client.ps1 use the pollinterval to know when splunk last ran .\bin\scom_client.path. so these should be same.

0 Karma

officeguy
Explorer

Right now I'm in testing mode. Local install of splunk on windows box which also has the scom console components installed. powershell query works fine and retries alert data when run manually in a powershell session. scom server is remote to this computer. Once I get it working then I'll move to install a splunk forwarder on the scom server, but for now splunk is local and scom is remote. I could always just change the powershell script to write each alert to a file which splunk could index.. but I still think I'm missing something in the initial app config..

0 Karma

bmacias84
Champion

My next question is are you running this remotely from a splunk instance or locally with a forwarder? Another question what is your deployment method for pushing out new configs, deployment server? The document is a little unclear, but I think i know whats going on.

0 Karma

officeguy
Explorer

sorry, I should have mentioned I am trying to get the scom app working: "http://splunk-base.splunk.com/apps/22380/scom-system-center-operations-manager-integration". All is set up fine, the powershell queries work, logging is ok, but no data reaches splunk. In reverse engineering the scom_client.ps1 file. the only 'forwarding' that happens is a write-host of the data to the powershell console. There seems to be a gap.. i.e it is unclear how splunk would ever notice this write-host unless it monitors this output by default by virtue of splunk calling the powershell session.

0 Karma

bmacias84
Champion

@officeguy, Can you tell how your ps1 script is ran? Are you using that in as a scripted input or someother method?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...