Deployment Architecture

No _internal results from distributed search head

_smp_
Builder

As a pretty new user, I recently installed the Universal Forwarder on a Linux server, created a file input, and forwarded to an indexer. This was working fine. Then as a result of a support case, I had to change the role from a UF to a Search Head in Distributed Search. After doing this and configuring the SH to forward its logs to the indexer, I am unable to return any results with a simple index=_internal search. Yet I can get results from all the non-internal indexes just fine. I have another SH (non-clustered) that works, and I have closely compared the Roles, but found no differences.

After searching the forum, I found a number of references to outputs.conf - here's mine:

[indexAndForward]
index = false

[tcpout]
defaultGroup = indexer
forwardedindex.filter.disable = true
indexAndForward = false

Not sure what else to look for?

0 Karma
1 Solution

lguinn2
Legend

Did you also set the search head to search the indexers?

By default, only a user with Admin privileges can see the internal indexes (_internal, _audit, etc.)
Check the role and make sure that these indexes are allowed to be searched.

View solution in original post

lguinn2
Legend

Did you also set the search head to search the indexers?

By default, only a user with Admin privileges can see the internal indexes (_internal, _audit, etc.)
Check the role and make sure that these indexes are allowed to be searched.

_smp_
Builder

The admin role was the problem. On my SH that was working, the admin role was restricted to internal and non-internal indexes. The SH which was broke, there were no selected indexes. After I added both internal and non-internal indexes to the admin role, I get the search results I was expecting.

Thank you.

This dialog box is confusing to me though. What is the default for the admin role - internal/non-internal, or nothing? The description for the dialog box explicitly states "Restrict", so my assumption is that if no indexes are selected, then there no restrictions. Am I wrong about this?

0 Karma

lguinn2
Legend

On the role, there are two index settings: indexes that the role is allowed to see ("Indexes" at the very bottom) and indexes that are default.

I can see how you might be confused by the language, but your assumption is wrong.
ONLY the indexes that appear in the "Indexes" list can be searched. Either an index must be explicitly chosen, or you can choose "All internal indexes" or "All non-internal indexes".

The "Indexes searched by default" must be a subset of that final "Indexes" list.

For the admin role, I usually set it like this -
Indexes searched by default = All non-internal indexes
Indexes = All non-internal indexes AND All internal indexes

_smp_
Builder

Thank you for your clarification.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...