Why do searches on the search head not return all data like searches run on indexers after distributed search setup?

[att35]

Hi,

We recently implemented distributed search by adding all existing Splunk enterprise servers as search peers. Setup went fine and we are able to see some data from the search head.

Problem is, not all data is visible. e.g. We are using OSSEC as one of the inputs to the indexer, but the search done on the head, doesn't return the sourcetype ossec_alerts. Instead, it returns a lot of data for sourcetypes (audittrail, splunkd_remote_searches and stash).

Same happens for a lot of searches. Data received is very different when compared with a search on the indexer itself.

Could it be just a replication issue? Or are we missing something in indexer setup? Only configuration was to add "search peer". Status is "Up" and replication status is "Successful".

Also, the user initiating the search is part of the role having access to all available indexers.

Many Thanks,

Abhi

[att35 · Accepted solution]

Thanks for the responses.

This has been fixed now. There were two issues.

1. The index being used by OSSEC on the individual indexers was _main. This was never an issue earlier because all searches / dashboards were local, but now, since the results were coming to a common search head, it started causing problems. To fix it, I re-directed the OSSEC to unique indexes, e.g. OSSEC_Site1, OSSEC_Site2, and then added these new indexes to the role on the search head.

2. I forgot to install the OSSEC app on the search head. 🙂 Although the first change fixed the data visibility issue, search head was still not able to parse the results.[ I was under the impression that since the parsing has already been done on the indexer, head would just pull the results]. Installing OSSEC app on search head resolved that issue.

Many Thanks once again for all the help

~ Abhi

[ppablo]

Hi @abhijittikekar

Glad you were able to solve your issue 🙂 Please be sure to accept your answer by clicking on the big check mark next to your response to mark this post as solved.

Happy distributed searching!

Patrick

[somesoni2]

Could you provide the searches that you're running (on both Search Head and Indexer) which are producing different results?

[ppablo]

Hi @abhijittikekar

I wasn't sure if you checked this out yet, but since you switched to a distributed search environment, it might be worth looking into. In the documentation tab under "Data Inputs" on the OSSEC app page http://apps.splunk.com/app/300/ inputs are disabled by default and in order to monitor OSSEC alert logs (ossec_alerts), Splunk has to be installed on the OSSEC server. Hopefully this helps with the missing OSSEC sourcetype and an expert on configuration will come along and figure out your issue 🙂