Alerting

How to monitor and capture muliple scenarios in same alert

Nidd
Path Finder

I have a requirement to monitor the below exceptions and send an alert through mail with few fields mentioned below.

Since I'm not able to achieve this, I have created 4 individual alerts and have monitored this. But that isn't right. I wish to capture all these within the same alert.

Below are sample logs.

 

TYPE 1: INVALID USERNAME/PASSWORD

 

 

2021-03-01 03:36:02,233 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR HikariPool - HikariPool-7333 - Exception during pool initialization.
java.sql.SQLException: ORA-01017: invalid username/password; logon denied

 

 

 

TYPE 2: INVALID SERVICE

 

 

2021-03-01 04:18:26,910 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

 

 

 

TYPE 3: INVALID PORT

 

 

2021-03-01 04:43:12,985 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1
validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection

 

 

 

TYPE 4: INVALID HOST

 

 

2021-03-01 05:02:13,113 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipelin-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified

 

Below are the fields to capture:

pipeline - Which is : my-pipeline-name

Exception -
Which are :
1. invalid username/password; logon denied,
2. TNS:listener does not currently know of service requested in connect descriptor
3. The Network Adapter could not establish the connection
4. Unknown host specified

 

Please help in achieving this.

Labels (2)
0 Karma

Nidd
Path Finder

Searches I use? Sorry. Didnt get you.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The searches used in the four alerts you already have

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What is wrong with having different alerts for the different conditions?

What are the searches you currently use?

0 Karma

Nidd
Path Finder

@ITWhisperer ..Yes..no issues in having 4 alerts.. But since we get the events from same server and same transaction, monitoring all the events in the same alert itself was what we have been given as a requirement.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK so combine the searches into one using OR conditions where appropriate.

What are the searches you are currently using?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...