Hey Splunk Community 🙂 Ok Ive got a tale of woe, intrigue, revenge, index=_*, and python 3.7 My tale begins a few weeks ago when myself and the other Splunk admin where just like "Ok, I know searches can be slow but like EVERYTHING is just draggin" We opened a support ticket, talked about it with AOD, let our Splunk team know, got told we might be under provisioned for SVCs and indexers no wait over provisioned, no wait do better searches, no wait again skynet is like "why is you instance doing that?". We also got a Splunk engineer assigned to our case and were told our instance is fine. Le sigh, when I tell you I rabbled rabbled rabbled racka facka Mr. Krabs .... I was definitely salty. So I took it upon myself to dive deeper then I have ever EEEEEVER dived before... index=_* error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ) I know I know it was a rough one BUT down the rabbit hole I went. I did this search back as far my instance would go. October 2022 and counted from there. I was trying to find any sort of 'spike' or anomaly something to explain that our instance is not fine. October 2022 -2 November 2022- 0 December 2022- 0 January- 25 February- 0 March- 29 April- 15 May-44 June- 1843 July-40,081 August- 569,004 September-119,696,269 October - dont ask, ok fine, so far in October there are 21,604,091 The climb is real and now I had to find what was doing it? From August and back it was a lot of connection/time out errors from the UF on some endpoints so nothing super weird just a lot of them. SEPTEMBER, specifically 9/2/23 11:49:25.331 AM This girl blew up! The 1st event_message was... 09-02-2023 16:49:25.331 +0000 ERROR PersistentScript [3873892 PersistentScriptIo] - From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-Zscaler_CIM/bin/TA_Zscaler_CIM_rh_settings.py persistent}: WARNING:root:Run function: get_password failed: Traceback (most recent call last): The rest of the event messages that followed were these ... see 3 attached screen shots I did a 'last 15 min" search but like September's show this hits the millions. Also, I see it's not just one app, its several of our apps that we use API to get logs into Splunk with, but not all the apps we use shows on the list (weird), and it's not just limited to 3rd party apps, the Splunk cloud admin app is on there among others (see attached VSC doc) I also checked that any of these apps may be out of date and they are all on their current version. I did see one post on community (https://community.splunk.com/t5/All-Apps-and-Add-ons/ERROR-PersistentScript-23354-PersistentScriptIo-From-opt-splunk/m-p/631008) but there was no reply. I also 1st posted on the Slack channel to see if anyone else was or had experienced this happening. https://splunk-usergroups.slack.com/archives/C23PUUYAF/p1696351395640639 and last but not least I did open another support ticket so hopefully I can give an update if I get so good deets! Appreciate you 🙂 -Kelly ... View more

Hi team 🙂 I have a user that left the company and now their dashboard searches are alerting as "orphaned objects". I reassigned all of their objects to me, cloned their dashboards (scream test), but when I go to (settings > user interface > views) to delete them I see no delete option except for the clones I made.  I changed the permissions on the dashboards to read/write the sc_admin role only  I (Admin) own all the objects now These dashboards were user made and not apart of 3rd party app What am I missing? I have a few screen shots below to show better what I am explaining Screen Shot of 'views' The clones are private the originals are not screen shot of what permissions are for original object I want to delete sc_admin has all the capabilities it can have assigned to it ... View more

I am running into an issue where nested AD groups that are in my Splunk AD group do not get the access that everyone else does. The situation when something like... I set up an AD group called Splunk_Win and there were several users in it who had the correct access and could view data. I had a manager request him and his team be added to the group so our Sysadmin added their team group to Splunk_Win and not individually. The manager then said they were getting error logging in and needed access now for an emergency. Our sysadmin decided it was best to just add the manager to Splunk_Win and whala, manager had the access he needed. I re-created this with another member of the group and asked them to screen shot what they saw (I can't add it but I'll type it out) Sorry, but we're having trouble signing you in AADSTS50105: The signed in user user.user@company.com is not assigned to a role for the application a1c025ed-e585-42ab-b809-a4f7b4fd3ea1 (Splunk Enterprise and Splunk Cloud. This error leads me to believe there is a disconnect between Azure and Splunk. The set up is SSO/SAML and as I said above, if the user goes into the Splunk AD group by themselves they get the access need. Has anyone run into this or has any ideas (besides adding individuals) to get nested groups to work in Splunk? ... View more

Hi there 🙂 I am in the process of migrating from Enterprise to Cloud and as I am setting up Splunk roles for the SAML groups that will use Splunk I am running into the issue that I have some elevated users that need to see certain index's but not everything. I wrote out an example below to get the user Tony the access he needs. • Lets say I have a SAML/AD group called Splunk_Marvel with Tony, Steve, and Peter as members. This group is to see the role “marvel events” and only this role (The role can see indexes called “comic” and “sidekicks”). • I also have a SAML/AD group called Splunk_DC with Bruce, Clark, and Harley as members, this group is only supposed to see the role “dc events” and only this role (this role can see indexes “comic”, “identity” and “powers”). • Now Tony asked his manager/put in a request to see the index “identity” and his manager approved but reminded him that only he can see it and not Steve and Peter. So what I’m running into for options/solutions to this is… • Give Splunk_Marvel the role “dc_events” …but now Steve and Peter are not supposed to see the index “identity” and Tony shouldn’t see “Powers” • Create a new SAML/AD group that just has Tony in it, add it as a SAML group in cloud, and also create a new role to view just the index “identity”. • Create a local spunk account for Tony to use (he’s a big enough deal) and assign that index to him specifically. WAIT!! wait wait Tony’s boss was talking to the sysadmin for Splunk and they said NO LOCAL SPLUNK ACCOUNTS!! ONLY SAML CAN BE USED FOR LOGIN!! So how do I give Tony the access he needs without creating a new AD/SAML group for just him or giving him access to data he and his team shouldn’t see. ... View more