Security

Why is our custom admin role unable to search _internal index data in Splunk Cloud?

hettervik
Builder

Hi,

On Splunk Cloud, the admin role has by default access to all non-internal indexes. At a customer's site, we want to retain the the access rights of admins by keeping "available indexes" (with the exception of internal indexes) in their own "access roles." Thus, our admin role would only grant the user the capabilities of the admin, but not any non-internal indexes. Those indexes would have to be granted by their own 1:1 access roles.

Our problem is that when we grant a user the custom admin role we've created, that user can't see the graphs in the "distributed management console." It seems like the users aren't allowed to see the _internal index, even though it clearly says in the custom admin role we've created that all internal indexes should be searchable. We've even added the index _internal specifically, in addition to "all internal indexes," just to be sure. What's interesting is that if we let our custom admin role inherit from the default admin role, then it works.

Any idea of what could be the problem here? Could there be some sort of "hidden" access right or capability on the default admin role in Splunk Cloud?

0 Karma
1 Solution

hettervik
Builder

Okay, I have an open case at Splunk Support on this issue, but haven't had any activity on the case for about three weeks now. Looks like it has come to a dead end. After some discussions with several colleagues we've come to the conclusion that there are some "hidden" capabilities for the default Splunk roles, meaning that you can't just copy the capabilities from e.g. the default admin role and create a custom admin role with the same capabilities but different indexes. Sorry, this is the best answer I can give. Case closed.

View solution in original post

0 Karma

hettervik
Builder

Okay, I have an open case at Splunk Support on this issue, but haven't had any activity on the case for about three weeks now. Looks like it has come to a dead end. After some discussions with several colleagues we've come to the conclusion that there are some "hidden" capabilities for the default Splunk roles, meaning that you can't just copy the capabilities from e.g. the default admin role and create a custom admin role with the same capabilities but different indexes. Sorry, this is the best answer I can give. Case closed.

0 Karma

mcmaster
Communicator

We just encountered this issue, not with an admin role, but with a role configured for some internal monitoring uses (but still on Splunk Cloud). Edit your role, and you should see that in the "Restrict search terms" box, it's filled in with "index!=*_archive". For whatever reason (and I really think this is a bug), when you run this search (even as an admin), you get zero results:

index=_internal index!=*_archive

You'll notice the admin role has "*" in the search terms. When we removed the search terms restriction on the custom, limited role, we were able to search the _internal index.

Hope this helps you too. Maybe jbailey can confirm if this is a bug.

hettervik
Builder

Thanks for your answer. I tried searching for index=_internal index!=*_archive while being logged in as the default admin, and the search did indeed return no results. Then I tried removing the archive search restriction from from my custom admin role, but I still can't search internal indexes. I also removed the same search restriction from my custom power role and my custom user role from which the custom admin inherits, again to little help. For me it's starting to look like we've found a bug.

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

I did not find any documentation stating that this is a bug, but I will follow up with others just to make sure.

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

It sounds like there is some difference(s) between the out of the box admin role and the custom admin role. Here's my suggestions to review/change and these are all under Settings->Access Controls->(Custom Admin Role):

1) Under -> Indexes Searched by Default
- Make sure "All Internal Indexes" is selected

2) Under -> Indexes
- Make sure "All Internal Indexes" is selected

If these do not resolve your issue, check the list of capabilities between the original admin role and your custom admin role - if any capabilities are missing when comparing against the original admin role, add those missing capabilities to the custom admin role.

0 Karma

hettervik
Builder

Hi. Thanks for your answer! I was thinking the same thing, there must be a difference between the two admin roles. I've triple checked that all capabilities from the default admin are copied over to my custom admin, also regarding inheritance from user and power roles. The custom admin have access to "all internal indexes," and "all internal indexes" are searched by default.

To me it almost seem like other roles than the default admin role on Splunk Cloud aren't allowed to search on internal indexes, even though the role specifies that they should be able to do so. Please, prove me wrong. I'd like to get to the bottom of this.

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

So, currently a user under the custom admin role is unable to see the graphs in the DMC - but if they execute a search of "index=_*", do they see results?

0 Karma

hettervik
Builder

No, my custom admin role don't see any internal data at all, e.g. index=_internal.

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

Are you receiving "No results found" after the search, or nothing at all? Can that user search any data/index at all?

0 Karma

hettervik
Builder

I receive "no results found" when searching for internal indexes. Searching for non-internal indexes works just fine.

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

Ok, thanks for your prompt response on that...

So, I tested your scenario out in a Splunk Cloud instance - creating a custom admin role, assigning no inheritance, assigning specific capabilities (1-for-1 match of default admin role capabilities), and selecting "All internal indexes" under "Indexes searched by default" and "Indexes" sections. Then, I created a new user and assigned that user the custom_admin role.

After saving it, I was able to login as the user assigned to the custom_admin role and successfully receive results from a search of "index=_*".

I'm posting my authorize.conf below (kinda lengthy) - compare this list to your authorize.conf, add any capabilities missing from your custom admin role to your role. You should be able to find yours under $SPLUNK_DB/etc/system/local/authorize.conf.

Also, if this doesn't work for you, you may want to at least setup inheritance from the "user" or "power" role.

[role_custom_admin]
accelerate_datamodel = enabled
accelerate_search = enabled
admin_all_objects = enabled
can_own_notable_events = enabled
change_authentication = enabled
create_external_ticket = enabled
cumulativeRTSrchJobsQuota = 400
cumulativeSrchJobsQuota = 200
edit_correlationsearches = enabled
edit_deployment_client = enabled
edit_deployment_server = enabled
edit_dist_peer = enabled
edit_forwarders = enabled
edit_httpauths = enabled
edit_input_defaults = enabled
edit_log_review_settings = enabled
edit_modinput_threatlist = enabled
edit_modinput_web_ping = enabled
edit_monitor = enabled
edit_notable_events = enabled
edit_per_panel_filters = enabled
edit_postprocess = enabled
edit_reviewstatuses = enabled
edit_roles = enabled
edit_scripted = enabled
edit_search_head_clustering = enabled
edit_search_scheduler = enabled
edit_search_server = enabled
edit_server = enabled
edit_sourcetypes = enabled
edit_splunktcp = enabled
edit_splunktcp_ssl = enabled
edit_suppressions = enabled
edit_tcp = enabled
edit_token_http = enabled
edit_udp = enabled
edit_user = enabled
edit_view_html = enabled
edit_web_settings = enabled
embed_report = enabled
get_diag = enabled
get_metadata = enabled
get_typeahead = enabled
indexes_edit = enabled
input_file = enabled
license_edit = enabled
license_tab = enabled
list_deployment_client = enabled
list_deployment_server = enabled
list_forwarders = enabled
list_httpauths = enabled
list_search_head_clustering = enabled
list_search_scheduler = enabled
output_file = enabled
pattern_detect = enabled
request_remote_tok = enabled
rest_apps_management = disabled
rest_apps_view = disabled
rest_properties_get = disabled
rest_properties_set = disabled
restart_splunkd = enabled
rtSrchJobsQuota = 100
rtsearch = enabled
run_debug_commands = enabled
schedule_search = enabled
search = enabled
srchDiskQuota = 25000
srchIndexesAllowed = ;_
srchIndexesDefault = ;_
srchJobsQuota = 50
srchMaxTime = 0
srchTimeWin = 0
web_debug = enabled

0 Karma

hettervik
Builder

Thank you for checking this out for me! How do you see your authorize.conf while using Splunk Cloud? As far as I know the only way to do it is to send a request to support. I will do so, and then I'll cross-check my configuration with yours. I'll get back to you.

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

You're correct that you don't typically have access to the authorize.conf within Splunk Cloud, and I forgot about that as I sent my previous post - sorry about that. The one I have access to is a unique situation, so typically no access is available.

You can compare what I sent you against your custom role within the UI: Settings -> Access Controls ->

Note: The srchIndexesAllowed and srchIndexesDefault in the listing above relates to the indexes sections at the bottom of that screen, in which mine has access to All Internal Indexes and All Non-Internal Indexes for both.

0 Karma

hettervik
Builder

I haven't gotten a copy of authorize.conf from support, but I've looked over the list of settings on my custom admin role in the Splunk Cloud GUI. The only "enabled" settings in your list that aren't in my UI list in our Splunk Cloud are the ones listed below. All these settings are not choosable from "available capabilities" in the Splunk Cloud role edit settings. Could any of these settings be the reason why my custom admin role can't see internal indexes? If so, do you know how to enable them?

can_own_notable_events = enabled
create_external_ticket = enabled
edit_correlationsearches = enabled
dit_log_review_settings = enabled
edit_modinput_threatlist = enabled
edit_modinput_web_ping = enabled
edit_notable_events = enabled
edit_per_panel_filters = enabled
edit_postprocess = enabled
edit_reviewstatuses = enabled
edit_suppressions = enabled

0 Karma

jbailey_splunk
Splunk Employee
Splunk Employee

You likely won't get a copy of the authorize.conf from the Cloud team, but you were able to compare the list I provided in the UI. I don't believe any of these settings you provided would impact the ability to see internal indexes.

At this point, I would suggest setting up inheritance from either the "user" role or "power" role, or possibly both - which is similar to the way the admin role is setup.

0 Karma

hettervik
Builder

I have sat up inheritance already. My custom admin role inherits from a custom power role and a custom user role. The custom power role and custom user role are copies of the default power and user roles in the same way the custom admin role are a copy of the default admin. In other words, this kind of inheritance setup doesn't seem to solve the problem. I've created a case on Splunk support. I'll update this thread if we find a solution.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...