Search head and indexer connectivity

[Mag2sub]

We have set up alerting searches with continuous scheduling from a search head with 2 peers
Soemtimes the search head loses connectivity with one of the peers
In this circumstance how does continuous scheduling work ..if it misses connectivity with one peer during a alerting search ...how do we safeguard against such circumstances ?

Apprecate inputs

[MuS]

Hi Mag2sub,

if your scheduled saved searches are configured to send alert emails you will get an email containing something like this:

-- Search generated the following messages -- 
Message Level: WARN
1. Unable to distribute to peer named <index-server> at uri https://<index-server>:<someportnumber>; because .....

In this case you would know about the condition you are describing.
You can also check out this answer the see a way to sent this kind of alert to a different recipient 😉

Best practice on the other hand would be to eliminate your cause for the errors.

hope this helps ...

cheers, MuS

[Mag2sub]

$6 just hits the jobs results endpoint ..which does not indicate any error neither $8 ...

[MuS]

ok, did you test SPLUNK_ARG_6 and / or SPLUNK_ARG_8 within your script? ARG_6 is an URL which could be loaded by your script and the parse the results or ARG_8 is the file system link to the result file which can be read by your script and be parsed for any errors.

[MuS]

pls hold the line and let me do some research. I know there is a way to do this as well in scripted alerts 😉

[Mag2sub]

We have a scripted alert and it does not get these from splunk ..it gives normal output ie normal arguments are passed and no error ...so i dont think this works for us

[linu1988]

status will always be success, if you can't get an instance or reproduce how do you want to test the alert? More often than not you will get an banner on splunkweb if there is an disconnection to search peer. then you know there must be some connectivity or indexer issue.

[Mag2sub]

Unfortunately there is no error in the scheduler for scheduled search ...its says staus=success which is mileading

[linu1988]

yes it will help if you have any error during the search. Use SOS app for monitoring the error trigger an alert. If it happens during a search it can't be avoided.

[Mag2sub]

1 Im looking for a way to safeguard against idx to SH connectivity loss during search time...how does a search work in that context

2 Connectivity loss message can be seen in the idx that was disconnected to SH
im looking to see how we can ensure seraches are not incomplete because of the loss of idx--sh connection..does continuous scheduling help ?

[linu1988]

how do you come to know that? Does it show in the search result in the mail?