I just stumbled on this and thought I'd add a few other notes on this... with props.conf in the stanza "sourcetype" where this props.conf creates a field called "action". Just to confirm, how is the field being created? I'm assuming you mean a search time field as opposed to an index time field. Skimming the #Mimecast for Splunk app it looks like there are field aliases, and eval statements for different source types around the `action` field which are both search time... but I could be missing something. If you were instead referring to an index time transformation, not only is precedence order different, but also reingestion of data would need to happen before things take effect. Speaking of precedence order, probably time to mention that search time attributes use user / app precedence order as documented: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles#Precedence_within_app_or_user_context  Some effects of this is 1) your app needs to be lexographically after the app you're trying to override (precedence by name of other apps is reverse lexographic order as opposed to forward lexographic order) 2) your app needs to export the corresponding settings 3) even with all that done, your settings from other apps will lose if searches are being launched from within the mimecast app because the current app's settings gets highest precedence for the same stanza. (may or may not be an issue, I'm not familiar enough with the app to say, but it is a potential edge case).  Here's where we need to address something else:  Splunk Cloud doesn't allow using local folder Splunk Cloud doesn't let you upload an app with a local folder, however, Calculated fields, and field aliases are both editable via the UI, so you could actually create a local override within the context of the original TA itself via the UI and those would win. (since merging between default and local per app happens first with user/app context resolution order). No additional app needed necessarily (that gets into a long term management discussion).  @richgalloway already provided an alternative solution since within props.conf, there is an additional merging between stanzas for sourcetypes, hosts, and sources... But if course need to be careful with those since you could affect other sourcetypes too. From the props.conf.spec: **[<spec>] stanza precedence:** For settings that are specified in multiple categories of matching [<spec>] stanzas, [host::<host>] settings override [<sourcetype>] settings. Additionally, [source::<source>] settings override both [host::<host>] and [<sourcetype>] settings. In either case once settings are in place on the search head, they need to be replicated to your indexers as part of the knowledge bundle before they can take effect during a search... So if you're already over the 3GB limit there need to spend some time trimming the bundle size.  Seeing resolved search time precedence can be done per stanza with the properties rest endpoint on the search head, and/or the btool command. (make sure to specify the appropriate `--app` and `--user` context for correct resolution order of search time values).  (And before you say but btool is an enterprise only command.. I may have brought it to Cloud as a SPL command along with a knowledge bundle utility in Admins Little Helper for Splunk ... my officially unsupported but I think its useful side project Check it out on Splunkbase: https://splunkbase.splunk.com/app/6368 </shameless plug> )  Hope these notes help you and others in the future. ... View more

Good to know about the different versions, thanks for the help! ... View more

Hi I am using something like this: index=_internal AND sourcetype=splunkd AND TERM(New) AND TERM(Old) AND TERM(properties) AND TERM(are) | rex field=_raw "Old properties are: \{ip=(?<old_ip>[^\ ,]*)[\ ,]+dns=(?<old_dns>[^\ ,]*)[\ ,]+hostname=(?<old_hostname>[^\ ,]*)[\ ,]+deploymentClientName=(?<old_deploymentclientname>[^\ ,]*)[\ ,]+.*instanceId=(?<old_instanceid>[^\ ,]*)[\ ,]+instanceName=(?<old_instancename>[^\ ,\}]*)" | rex field=_raw "New properties are: \{ip=(?<new_ip>[^\ ,]*)[\ ,]+dns=(?<new_dns>[^\ ,]*)[\ ,]+hostname=(?<new_hostname>[^\ ,]*)[\ ,]+deploymentClientName=(?<new_deploymentclientname>[^\ ,]*)[\ ,]+.*instanceId=(?<new_instanceid>[^\ ,]*)[\ ,]+instanceName=(?<new_instancename>[^\ ,\}]*)" | stats latest(_time) AS ltm, count BY new_hostname, old_hostname, new_dns, old_dns, new_instancename, old_instancename, new_instanceid, old_instanceid, new_ip, old_ip | convert timeformat="%y-%m-%d" ctime(ltm) AS LastDay | fields - ltm   ... View more

Are you certain you had the :8089 as part of your curl url? AND you used the correct url? The redirection response you have provided is identical to the one that Splunk Web (i.e. port 443 OR no port specified with HTTPS) would give in response to a request for /servicesNS/nobody/search/data/indexes (Which would be the enterprise API url instead of the cluster blaster one you state in your post.) Deliberately omitting the :8089 from the cluster_blaster_indexes request against my classic stack I get the following: $ curl https://redacted.splunkcloud.com/services/cluster_blaster_indexes/sh_indexes_manager?output_mode=json <!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta http-equiv="refresh" content="1;url=https://redacted.splunkcloud.com/en-US/services/cluster_blaster_indexes/sh_indexes_manager?output_mode=json"><title>303 See Other</title></head><body><h1>See Other</h1><p>The resource has moved temporarily <a href="https://redacted.splunkcloud.com/en-US/services/cluster_blaster_indexes/sh_indexes_manager?output_mode=json">here</a>.</p></body></html>   ... View more

I like this query, but I have indices with long names that incorporate underscores "_" and the split command is not working in this scenario. Quotations did not work, but astericks did.  I do not want to use asterisks as I will be generating alerts and do not want extra characters in the message.  Please let me know how to use split with this naming convention for indices.  | metasearch index=AB_123_CDE OR index=CD_345_EFG OR index=EF_678_HIJ | stats count by index | append [ noop | stats count | eval index=split("AB_123_CDE;CD_345_EFG;EF_678_HIJ",";") | mvexpand index ] | stats max(count) as count by index | where count = 0 ... View more

Hi @yeahnah ...though you gave the answer first, the other user gave the detailed explaination..  i have given upvote for your post. hope you understand my view, thanks.  ... View more

Thanks @acharlieh your recommendation did the trick what I needed now. Yes, your assumption is right the table comes from stats command, I will try to play with it more, but for now the first solution helped, thank you so much 🙂 ... View more

Thanks, @acharlieh , this is awesome.  Exactly what I needed, fed to me on a spoon.  Really appreciate the time you saved me. ... View more

It is a nice way but it works great only if you're copy-pasting from somewhere. If you're producing data by hand this way, you need all this spaces counting. I'd rather do a simple json and then spath/mvexpand if needed. But I usually generate data with something like for example | makeresults count=100 | streamstats count | eval fieldA=count % 12 | eval fieldB=count % 7 | [... and so on ...]   ... View more

Thanks, this was a misread of the RFC on my part.  I appreciate the help. ... View more

You probably want to reach out to the developer of the custom addon to help with troubleshooting, if they still support it. Quick search for prtglivedata and I found this app: https://splunkbase.splunk.com/app/3282 which is only marked as being supported on Splunk 7. (Which is of course End of life, and before the Python3 transition), but I'm not sure if thats your app or not. These conf file configurations by themselves are slightly odd since they have both  chunked = true and arguments that only make sense for the original Intersplunk protocol. From commands.conf.spec chunked = <boolean> * Whether or not the search command supports the new "chunked" custom search command protocol. * If set to "true", this command supports the new "chunked" custom search command protocol, and only the following commands.conf settings are valid: * 'is_risky' * 'maxwait' * 'maxchunksize' * 'filename' * 'command.arg.<N>' * 'python.version', and * 'run_in_preview'. * If set to "false", this command uses the legacy custom search command protocol supported by Intersplunk.py. * Default: false   For reference... the "new" protocol came about a really long time ago. (Splunk SDK changes to support chunked were back in 2015 so something like Splunk 6? )  You could try to look in the search log (see the job inspector), splunkd log, or python log (index=_internal) to see if any errors or stacktraces related to your script are emitted.  ... View more

Something broke the rendering of the markup here ...??? ... View more

Please find the below link to find the version 8.2.10 https://www.splunk.com/en_us/download/previous-releases.html ... View more

If there isn't a boot event (i.e. an event with the words "Linux version" in it) for a particular host in your time window, boot_time will come back as blank...  This is the problem I was mentioning: But the more practical problem you'll run into is the unbounded nature of how far in the past boot time can be... thus requiring this search to become almost an All Time search which doesn't scale well at all.    ... View more

Well I understand your point about "this"... but that's the problem, I couldn't find an error with the skipped searches... unless I am missing something. Since I did the rolling restart (reset) there are no more skipped searches. Previously I looked for the longest running searches and none were over-running their schedules, that I could see.  For example one search took an hour approx., but it ran every 4 hours.   Since I did some optimizing there were only 3 scheduled searches that produced the warning which I identified with    index="_internal" sourcetype="scheduler" | eval scheduled=strftime(scheduled_time, "%Y-%m-%d %H:%M:%S") | stats values(scheduled) as scheduled values(savedsearch_name) as search_name values(status) as status values(reason) as reason values(run_time) as run_time values(dm_node) as dm_node values(sid) as sid by _time,savedsearch_name | sort -scheduled | table scheduled, search_name, status, reason, run_time     When I looked back at those 3 specific searches, they were not over-running the schedules, so I was wondering how it got stuck thinking it was "piling up" vs "still running".      I am trying to understand/investigate, if a search is "skipped" then when the shc scheduler retries that previously skipped search at its next runtime, "how can I see that the shc CPT thinks its still running"?  And looking back at the "skipped" events, they don't contain "run_time"...  so I look back historically to find a day with a high value.  But when the searches were running they took max 4 seconds with avg of 2 seconds to complete, which is why I thought the scheduled searches were piling up.  Hope that makes sense. The only other variable I can think of is that these searches are using the "| dbxquery" cmd from Splunk DB Connect app. So did it the SHC just get stuck? Any further thoughts appreciated. TY     ... View more

To expand on what @dwaddle said… The ideal way to solve this would be to configure systemd to start the Splunk service with the CAP_DAC_READ_SEARCH capability, which bypasses file read permission checks and directory read and execute permission checks. Effectively, a process with CAP_DAC_READ_SEARCH can read any file or open and directory on the system. (Depending on how Splunk uses fanotify(7), it might also need the CAP_SYS_ADMIN capability.) Unfortunately, a bug in Splunk prevents this from working. Specifically, Splunk erroneously uses faccessat(2) to check whether it has read access to a log file before attempting to open it. Per access(2), when a non-root user calls any of access(2) family of system calls, the access test is performed using an empty capability set. Quoting the man page verbatim: In other words, access() does not answer the "can I read/write/execute this file?" question. It answers a slightly different question: "(assuming I'm a setuid binary) can the user who invoked me read/write/execute this file?", which gives set-user-ID programs the possibility to prevent malicious users from causing them to read files which users shouldn't be able to read. Splunk misunderstands this—it is using faccessat(2) to ask “can I read this log file?”, which is wrong; faccessat(2) will return EPERM even though opening the log file would succeed (because the Splunk process holds the CAP_DAC_READ_SEARCH capability). On a Unix system, the correct way to test whether the current process has read access to a file is simply to attempt to open the file. If the attempt succeeds, you have read access; if the attempt fails with EPERM, you don’t. Using faccessat(2) to test whether the current process has read access is wrong. I reported this to Splunk as a bug; Splunk refused to acknowledge it as such. The only option Splunk gave me to try to get this bug corrected was to file a Splunk idea, which I did: EID-I-759. If anyone else would like to see this long-standing bug in Splunk finally fixed, I would encourage you to vote for EID-I-759. ... View more

Wow! As simple as that! Thank you! Just to simplify. All I had to do was change my drilldown from this: To this: Now when I select a date column, my time picker changes! Thanks @acharlieh !! ... View more

So you might need to upgrade MLTK to 5.3.3, or follow the workaround to manually update conf files as listed in the Known issues under 5.3.1: https://docs.splunk.com/Documentation/MLApp/5.3.1/User/Knownissues ... View more

There is NO Edit Login and Email subscriptions option in My Dashboards. ... View more

As it comes out of the box, the Splunk Add-on for Cisco ESA has no UI components.... If it was there'd be a default/data/ui folder which is missing... and the app.conf even states this isn't a visible app:   $ tar tzvf splunk-add-on-for-cisco-esa_160.tgz drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/ drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/LICENSES/ -rw-r--r-- 1001/121 85947 2022-07-25 01:18 Splunk_TA_cisco-esa/LICENSES/LicenseRef-Splunk-8-2021.txt -rw-r--r-- 1001/121 165 2022-07-25 01:18 Splunk_TA_cisco-esa/README.txt -rw-r--r-- 1001/121 1916 2022-07-25 01:18 Splunk_TA_cisco-esa/THIRDPARTY -rw-r--r-- 1001/121 11 2022-07-25 01:18 Splunk_TA_cisco-esa/VERSION -rw-r--r-- 1001/121 1551 2022-07-25 01:18 Splunk_TA_cisco-esa/app.manifest drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/default/ -rw-r--r-- 1001/121 473 2022-07-25 01:18 Splunk_TA_cisco-esa/default/app.conf -rw-r--r-- 1001/121 4770 2022-07-25 01:18 Splunk_TA_cisco-esa/default/eventtypes.conf -rw-r--r-- 1001/121 24749 2022-07-25 01:18 Splunk_TA_cisco-esa/default/props.conf -rw-r--r-- 1001/121 1208 2022-07-25 01:18 Splunk_TA_cisco-esa/default/tags.conf -rw-r--r-- 1001/121 50633 2022-07-25 01:18 Splunk_TA_cisco-esa/default/transforms.conf drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/ -rw-r--r-- 1001/121 85 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_authentication_action_lookup.csv -rw-r--r-- 1001/121 617 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_email_action_lookup.csv -rw-r--r-- 1001/121 920 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_proxy_status_action_lookup.csv -rw-r--r-- 1001/121 309 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_vendor_info_lookup_160.csv drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/metadata/ -rw-r--r-- 1001/121 105 2022-07-25 01:18 Splunk_TA_cisco-esa/metadata/default.meta drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/static/ -rw-r--r-- 1001/121 3348 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIcon.png -rw-r--r-- 1001/121 3348 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIconAlt.png -rw-r--r-- 1001/121 6738 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIconAlt_2x.png -rw-r--r-- 1001/121 6738 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIcon_2x.png $ tar xOzvf splunk-add-on-for-cisco-esa_160.tgz Splunk_TA_cisco-esa/default/app.conf | grep visible Splunk_TA_cisco-esa/default/app.conf is_visible = false   (Lots of addons don't ship UI pieces, especially if they're only doing non-ui related things like setting up props and transforms, or lookup enrichments) Without knowing your stack or this app in depth, I suspect what is actually happening is that someone set the app to visible in your environment (which isn't needed), and more likely than not some other app is globally exporting its nav default.xml with a default view set that isn't being shared globally, thus when you open the app, you try to get to a view that isn't available and thus 404. (Using Settings > User Interface > Navigation Menus you can see if there's a nav bar visible in this app's context from a different app) Or there could be other quirks... but this add on has no UI components out of the box... ... View more

Outrageous trick!  Very helpful, thank you!  😀 ... View more

es, I did it myself, and I am currently running SPLUNK version 8.0.4. ... View more