I am in the process of writing a Splunk script that is going to overwrite the contents of a lookup file using REST. However, the issue I am hitting is how to authenticate against the REST endpoint. I am planning on having Splunk running the script ( probably through inputs.conf). It would every x hours and update the lookup using a python script that calls an outside source. I can successfully call the outside source and parse the data, however I am stuck on how to overwrite the lookup table via REST. All examples of REST calls show passing credentials. I dont want to hardcode any admin creds on the script itself. I found this article from splunk, but the REST section clearly shows they are passing creds. Are there any other ways to do this?  https://www.splunk.com/en_us/blog/tips-and-tricks/store-encrypted-secrets-in-a-splunk-app.html Any suggestions? ... View more

I have a requirement where I need to make an API call and write the data to a lookup file that I can use locally. The API calls returns data in a CSV format.    Previously, I used the Ad-on builder to create a python script that would make make the API request and index this data. However, I have a new requirement to skip the index entirely and write to a local lookup on the search head. The Ad-on builder wont help as it only shows examples of how to write the data to an index.   Thank you! ... View more

I am trying to create a new process to have a service (non-admin) account adding new Search Heads into a cluster. Specifically, need enough capabilities to to have the service account initialize and add a search head to a cluster. I want to avoid giving "admin-all-objects" as its too much privilege and want to adhere to least-priv policy.   I created a new local account and added it to the deployer, SH cluster, and the new SH. I then added capabilities related to SH clustering so that i can have this service account initialize and add SH to a cluster. However, i am getting errors related to permissions. Capabilities added: edit_restmap edit_search_head_clustering edit_search_server edit_server list_search_head_clustering rest_apps_management rest_apps_view rest_properties_get rest_properties_set restart_splunkd     Error when trying to initialize SH:    /opt/splunk/bin/splunk init shcluster-config <CLUSTER INFO>> User 'shcluster_config' with roles { shcluster_config, user-shcluster_config } cannot write: /nobody/system/server { read : [ * ], write : [ admin ] }, removable: no       Anybody know what capability i need to give this service account enough access to add new SHs to a cluster?   Thank you ... View more

Hi all,   I have an interesting problem I discovered. Recently, we migrated our Splunk Cluster to a different cluster hosted somewhere else. Since we use LDAP authentication , we need to migrate over User information as well as the LDAP strategies so that the user experience is not affected by the move. We copied over the authorize.conf, authentication.conf as well as the user folder for their KO. There were over 100 different users that we did this. We deployed the user folder using the new cluster's Deployer and we copied over the authorize.conf/authentication.conf manually to the system/local folder. We verified user access and various users were able to verify that they can login. However,  we  (the splunk Admins) realized that we cannot see these users logging in from the authentication endpoint. When we click the User tab under "Users and Authentication" in Settings, the GUI only shows that there are 10 users (including the admins). The rest endpoint ( |rest /services/authentication/users) also says the same thing.    So my question is,  where does Splunk store user information that it references when hitting the authentication endpoint ?  Is there any reason why copying over the User folder and authentication/authorization.conf was not enough?   Thank you! ... View more

I've been looking around how to measure and scale a Splunk HEC Collector cluster, but I cant seem to find direct answers. I am trying to find a way to measure the performance of a HEC Collector and how to determine when a HEC Collector cluster needs to be scaled to accommodate more/less HEC requests. I understand that EPS ( events per second) can be measured but how does one determine if that number is too high or too low? Currently, i have two Heavyforwarders that are acting as HEC Collectors behind a load balancer. I am trying to find out a good way to determine if this is enough or if another member needs to be added. Appreciate the help! ... View more

Hey y'all, So I am seeing a very unique and strange behavior from Splunk. I noticed an issue where a Splunk search returned an event if the time range was narrow, but would not return the same event if the time range was broader. For example, the following Splunk SPL would return 1 event when the time range was set to May 2nd to May 3rd. index=firewalls dest_ip=124.65.193.752 src_ip=10.42.68.21 However, the same SPL would return 0 events when the time range picker was set to "Last 90 days" index=firewalls dest_ip=124.65.193.752 src_ip=10.42.68.21 The last 30 days from today would essentially cover March 15th to June 15th (Today). We have a ton of Firewall logs so the only thing I can think of is that Splunk accidentally skips over the event due to the sheer volume of data....however no warning is thrown after the search is run. Splunk throws no warning or issues under the Job tab. Any ideas would greatly be appreciated! ... View more

I am trying to figure out how the Size value in the Job page is calculated and where that is logged in splunk. I checked: | rest /services/search/jobs|table label diskUsage The diskUsage value was close but didnt match the Size value in the Job page. In Introspection, index=_introspection | table _time data.search_props.user data.search_props.provenance data.elapsed data.mem_used data.search_props.user data.written_mb Data.mem_used and data.written_mb seem to be close ( after adding the values since a job creates multiple events) but they also dont match the Size value. Cant seem to find a way to locate that and create a report from it. I am basically trying to find a way to make a report showing user job history and the size of their jobs. Any help would be great! ... View more

I have a request to see if I can add a right click functionality to some of the dashboards. If right clicking on a table or row, a custom menu would pop up allowing us to link to other dashboards or search a URL, etc . In another question, I was referred to bootstrap library, but am not sure how to implement the actual syntax into the XML of the dashboard. ... View more