Why is the "user" role showing up on some single search head cluster member, but not the others?

[Lucas_K]

I've found that the default user group role is being applied to users on some search heads, but not others.

I can see this in the search log

Search head 1. Expected roles.

7-20-2016 04:34:47.755 INFO  dispatchRunner - Arguments are: "search" "--id=1468989287.51750_F7F40473-8726-4D4B-8160-4625565894FE" "--maxbuckets=300" "--ttl=600" "--maxout=500000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=10" "--rf=*" "--user=someuser" "--pro" "--roles=xxxxxxxx:std-admin:std-power:std-user"

Search head 2. Unexpected roles.

07-20-2016 04:38:03.125 INFO  dispatchRunner - Arguments are: "search" "--id=1468989482.13557_563FF8CD-6257-4804-A02F-F948DB01C42B" "--maxbuckets=300" "--ttl=600" "--maxout=500000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=10" "--rf=*" "--user=someuser" "--pro" "--roles=:std-admin:std-power:std-user:user"

We have explicitly created our own roles that do not use the included default "user", "power" or "admin" roles. We have instead created similar functioning ones called "std-admin/power/user".

Using a btool ( splunk btool authorize list --debug | grep import | sort -u ) there isn't any import of this role nor any other group that imports it into their own roles (apart from default) yet the user group shows up under my ldap authenticated account.

I've checked other apps such as dbconnect/dbx and they had already been modified so to not import the standard roles.

There is no ldap mapping to the user group so I am at a loss as to how it is picking up this default user role only on certain search heads.

Is there any other way I can track down how this is being applied?

[hardikJsheth]

Roles are defined in authorization.conf file and assigned using authentication.conf file. Both these files are available under auth application in $SPLUNK_HOME/etc/apps folder.

You can check content of this files to find out how roles are getting applied.

[Lucas_K]

You must have missed the bit in my post where I actually check that using btool debug (debug actually shows the file location of the specific setting location).

This shows that is isn't being applied ANYWHERE in the config yet the search.log shows that it is. The behaviour is that it is being applied also.

The real issue is that it exposes a VERY large security issue if you aren't aware of it. I found this on a newly commissioned yet not customer accessible search head. I was trying to understand why I was seeing data with a test account that I shouldn't.

[hardikJsheth]

What I meant was to compare authorize.conf file on both search head.

The command, that you are referring too, only displays how many times importRoles is present in your authorize.conf. If you don't see it in the output, that doesn't mean the role is disabled.

By default any user you create in Splunk is assigned "user" role if it exist. Just check the configuration of the file and you will find out why you have user role present in log.

[Lucas_K]

hmm. All my users (apart from local admin) account are all ldap. There isn't any config that maps these users to that role via either inheritance or directly. We deliberately changed changed it when the security hole was discovered (either late v5 or early v6).

The other issue is that all the authorize.conf files are identical as they are pushed out by the deployer that controls the search head cluster configuration. I've done a diff just to be sure and they all match.

"If you don't see it in the output, that doesn't mean the role is disabled."

This is why I was asking if there was another way to see it. I might hunt around the rest api and see if there is a way to query it.

Btool doesnt show the inheritance and the gui doesnt show the inheritance yet shows all the other attached role groups like normal.
The only place I can see it is in the search.log and by way if it showing me more data than I should be able to see.

[Lucas_K]

Found a rest api that shows me the issue.

Doing this from a utility server that can see across all search heads highlights the problem.

| rest /services/authentication/users/testaccount | stats values(roles) by title,splunk_server | search title=testaccount

The only problem is that it is showing me the current "state" and not where that difference comes from.

Running this on each server shows that only the default roles and none of my custom roles have the "user" role imported into them.

|localop | rest /services/authorization/roles | stats values(imported_roles) as importroles by title, splunk_server | mvexpand importroles | search importroles=user