Deployment Architecture

How can I set up the "Log Event" alert action in a distributed environment?

ctaf
Contributor

Hello,

I am trying to use the new alert action "Log event" in a distributed environment (Search Head 6.4.0 & Indexers 6.2.2).
Unfortunately, I doesn't work properly.

For the test, I set the "main" index as the destination index.

First issue: it seems that it is writing in "main" index, but on the Search Head, not on the Indexer (there is no way to indicate onto which Search peer to write the log by the way..)
Second issue: I cannot see the written log. When I search index=main, there is no result. I only guess that the event is written because when I go to the "Indexes" pages in the setting, the "Latest event" time is updated.

Any idea how to make it work?

0 Karma

emeelan_splunk
Splunk Employee
Splunk Employee

Hi folks,
This issue will be fixed in our next release.

Thank you for your patience.

-Eve Meelan

0 Karma

emeelan_splunk
Splunk Employee
Splunk Employee

Hi folks,
Strive's comment is correct. In order for Custom log alert events to be set up in a distributed environment, you must define the index on the search head. We are looking at this as a bug, but Strive's work-around is valid. For reference, the bug number is SPL-146802

0 Karma

pkarpushin
Path Finder

Hello,
Splunk 7.1.3 same issue

0 Karma

DATEVeG
Path Finder

Any news on fixing this?
I'm on 7.1.3 and the bug still seems to be present.

Thanks!

0 Karma

strive
Influencer

We faced similar issue.

The summarized data forwarding to indexers works fine. Other internal logs forwarding to indexers work fine. But, the Alert Action Log event alone fails.

For this to work -- We have to define index in Search Head as well.

Note: the data wont be stored on Search Head, eventually the Alert Action's Log event will be forwarded to indexer. But the definition on SH is must.

Not sure if it is a bug in Splunk or it is working as expected.

I have added a comment in Alert documentation and a discussion is on with splunk folks 🙂

nick405060
Motivator

Not very elegant. But fixes my problem. Thanks!

0 Karma

somesoni2
Revered Legend

Check why Search Head is not Forwarding the data to Indexers (it should forward instead of indexing locally). Check the outputs.conf. I believe second issue will resolve itself once you fix this.

0 Karma

ctaf
Contributor

Oh OK it makes sense for the first point. Unfortunately, I can't make this change due to my company policy.
But I should be able to search my local index on the search head, right? (second point) I do it for _internal index and there is no issue.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

REGISTER NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If ...

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...