Selective forwarding and summary searches - any issues?

[arkadyz1]

Hello,
I'm experimenting with some selective forwarding and it's mostly working - I can index locally, forward and combine both.

One strange occurrence is the fact that the searches which collected events into summary indexes stopped working. When I look at the recent searches, they report some non-zero numbers of events - but there is nothing new in the summary index!

In fact, the most recent event in the summary index is prior to the time when I added outputs.conf with indexAndForward stanza, and edited inputs.conf, props.conf and transforms.conf, adding all those TCP_ROUTING and _INDEX_AND_FORWARD_ROUTING where necessary.

Is there anything I'm overlooking? Any definition for summary index I have to add?

[Masa]

Not sure if summary index pipeline works for selective indexing because summary index pipeline has some hard-coded restriction.
All use cases I know are for true forwarders instead of search head.

You can try the following settings in inputs.conf

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
_TCP_ROUTING = <your tcpout value>

Or,

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_new]
_INDEX_AND_FORWARD_ROUTING =  <your tcpout value>

Sounds like you have standalone Splunk solution, instead of distributed search architecture.
In distributed search architecture, local indexing at search is not a good practice and should avoid as much as possible because search heads requires to do indexing jobs and search peer jobs when search head is indexing locally.

[arkadyz1]

Thanks, I will definitely try this.

We do have mostly standalone solutions, but with a twist: our customer is a huge company with many buildings spread across the country. Each building is overseen with a separate, standalone Splunk instance with the same applications installed. That application uses summary indexes heavily, both for efficiency and as to achieve uniformity among the incoming data, formed by several different vendors.

However, there are also some Splunk instances in the company's headquarters. They run different applications, which are used mostly for monitoring (but not just), and they need to process some of the data which are collected by those "branch" installs.

So both the "branches" and the "HQ" have to be search heads, and we'd like to have some events propagated from the branches to the HQ (while still being indexed there locally). And the summary searches to continue working.

As an alternative to selective forwarding, I was going to look at the HTTP event collector and make a custom alert action which sends the events that way. I'll need to understand more about it, though.

[Masa]

Thank you for sharing your use case.
Sounds like it is a challenge to maintain such standalone data independently, and forwarding partial data.
Good luck