I just stumbled on this and thought I'd add a few other notes on this... with props.conf in the stanza "sourcetype" where this props.conf creates a field called "action". Just to confirm, how is the field being created? I'm assuming you mean a search time field as opposed to an index time field. Skimming the #Mimecast for Splunk app it looks like there are field aliases, and eval statements for different source types around the `action` field which are both search time... but I could be missing something. If you were instead referring to an index time transformation, not only is precedence order different, but also reingestion of data would need to happen before things take effect. Speaking of precedence order, probably time to mention that search time attributes use user / app precedence order as documented: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles#Precedence_within_app_or_user_context  Some effects of this is 1) your app needs to be lexographically after the app you're trying to override (precedence by name of other apps is reverse lexographic order as opposed to forward lexographic order) 2) your app needs to export the corresponding settings 3) even with all that done, your settings from other apps will lose if searches are being launched from within the mimecast app because the current app's settings gets highest precedence for the same stanza. (may or may not be an issue, I'm not familiar enough with the app to say, but it is a potential edge case).  Here's where we need to address something else:  Splunk Cloud doesn't allow using local folder Splunk Cloud doesn't let you upload an app with a local folder, however, Calculated fields, and field aliases are both editable via the UI, so you could actually create a local override within the context of the original TA itself via the UI and those would win. (since merging between default and local per app happens first with user/app context resolution order). No additional app needed necessarily (that gets into a long term management discussion).  @richgalloway already provided an alternative solution since within props.conf, there is an additional merging between stanzas for sourcetypes, hosts, and sources... But if course need to be careful with those since you could affect other sourcetypes too. From the props.conf.spec: **[<spec>] stanza precedence:** For settings that are specified in multiple categories of matching [<spec>] stanzas, [host::<host>] settings override [<sourcetype>] settings. Additionally, [source::<source>] settings override both [host::<host>] and [<sourcetype>] settings. In either case once settings are in place on the search head, they need to be replicated to your indexers as part of the knowledge bundle before they can take effect during a search... So if you're already over the 3GB limit there need to spend some time trimming the bundle size.  Seeing resolved search time precedence can be done per stanza with the properties rest endpoint on the search head, and/or the btool command. (make sure to specify the appropriate `--app` and `--user` context for correct resolution order of search time values).  (And before you say but btool is an enterprise only command.. I may have brought it to Cloud as a SPL command along with a knowledge bundle utility in Admins Little Helper for Splunk ... my officially unsupported but I think its useful side project Check it out on Splunkbase: https://splunkbase.splunk.com/app/6368 </shameless plug> )  Hope these notes help you and others in the future. ... View more

Are you certain you had the :8089 as part of your curl url? AND you used the correct url? The redirection response you have provided is identical to the one that Splunk Web (i.e. port 443 OR no port specified with HTTPS) would give in response to a request for /servicesNS/nobody/search/data/indexes (Which would be the enterprise API url instead of the cluster blaster one you state in your post.) Deliberately omitting the :8089 from the cluster_blaster_indexes request against my classic stack I get the following: $ curl https://redacted.splunkcloud.com/services/cluster_blaster_indexes/sh_indexes_manager?output_mode=json <!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta http-equiv="refresh" content="1;url=https://redacted.splunkcloud.com/en-US/services/cluster_blaster_indexes/sh_indexes_manager?output_mode=json"><title>303 See Other</title></head><body><h1>See Other</h1><p>The resource has moved temporarily <a href="https://redacted.splunkcloud.com/en-US/services/cluster_blaster_indexes/sh_indexes_manager?output_mode=json">here</a>.</p></body></html>   ... View more

The general concept of a backreference is the same between both. Splunk's rex/regex processing in ingestion and during a search is powered by the Perl Compatible Regular Expressions library. There are syntactic and execution differences between PCRE & GNU SED's regular expressions, but other forums and sites would be appropriate for detailing out those exact differences. I think I have also heard that there may be some (recently released / soon coming) products/features that may leverage the RE2 regex library instead (partly from being Golang based and also some more control on predictability in time/complexity bounds of execution). Of course RE2 has it's own set of nuanced differences from PCRE & GNU SED, but general concepts are similar, and it too can be tested in regex101.com (pick the Golang flavor instead of the default PCRE) ... View more

You probably want to reach out to the developer of the custom addon to help with troubleshooting, if they still support it. Quick search for prtglivedata and I found this app: https://splunkbase.splunk.com/app/3282 which is only marked as being supported on Splunk 7. (Which is of course End of life, and before the Python3 transition), but I'm not sure if thats your app or not. These conf file configurations by themselves are slightly odd since they have both  chunked = true and arguments that only make sense for the original Intersplunk protocol. From commands.conf.spec chunked = <boolean> * Whether or not the search command supports the new "chunked" custom search command protocol. * If set to "true", this command supports the new "chunked" custom search command protocol, and only the following commands.conf settings are valid: * 'is_risky' * 'maxwait' * 'maxchunksize' * 'filename' * 'command.arg.<N>' * 'python.version', and * 'run_in_preview'. * If set to "false", this command uses the legacy custom search command protocol supported by Intersplunk.py. * Default: false   For reference... the "new" protocol came about a really long time ago. (Splunk SDK changes to support chunked were back in 2015 so something like Splunk 6? )  You could try to look in the search log (see the job inspector), splunkd log, or python log (index=_internal) to see if any errors or stacktraces related to your script are emitted.  ... View more

@sshubh People answering posts on here are doing so by donating their time. As such you should not be preemptively tagging people to help with your question. You'll attract responses by showing what you've tried, what result you got and how you thought it should be different, and showing a willingness to learn. You posted the exact same question a few days ago, and were given an answer. Instead of asking again with no additional information, you could follow up with what is not working for you or what you're not understanding with that answer. It may help to break down the problem into steps and check each one. Start with are your fields extracted from your events properly? Do you get all values of the multi-valued fields as you expected on each event? Are your field names lined up between the bought and sold records so you can correlate them together? Do you have a field marking a result as a bought or a sold transaction already? ITWhisperer did some field manipulation and extraction with eval.  Assuming you have the above done correctly, did you know that using makeresults and eval we can actually simulate your example data set with a Splunk Search and anyone can build from it? (names of fields might be slightly different, but this should be where you are at this stage.  | makeresults count=8 | streamstats count | eval AccountName=case(count in(1,2,6),"ABC", count=3,"DEF", true(),"EPF"), TransactionType=if(count<=5,"bought","sold"), BookId=case(count=1,split("book1,book2,book3",","),count in (2,5,7,8),"book1",count=3,split("book1,book2",","),count=4,split("book1,book3",","),count=6,"book2") | fields - _time count If you don't have the above done correctly, then anything afterwards isn't going to work, and you should be talking about that problem first. (I'll also note that field names are always case sensitive) From this point IT Whisperer already showed you how stats can group by multiple fields, and even showed you the trick with eval and french braces {} in order to create fields with names based on the values of other fields, and running stats multiple times to combine things down.  You can use the same tricks in a slightly different order to not need the fillnull command (but it's still useful to know).  | eval T_{TransactionType}=1 | stats count(T_*) as * by AccountName BookId | stats list(BookId) list(bought) list(sold) by AccountName   I leave the total books calculation as an exercise for you, but also the hint that stats can perform multiple statistical functions in a single pass on multiple different fields of the input data set. ... View more

If there isn't a boot event (i.e. an event with the words "Linux version" in it) for a particular host in your time window, boot_time will come back as blank...  This is the problem I was mentioning: But the more practical problem you'll run into is the unbounded nature of how far in the past boot time can be... thus requiring this search to become almost an All Time search which doesn't scale well at all.    ... View more

The key words there are "for this historical scheduled search"... So likely looking at a search job that's taking longer than its scheduled period to execute. I'd start with looking at the runtimes of the skipping search you've already found. (of course not ruling out something crazy like the job wasn't running but the SHC captain thought it was...)   ... View more

So for the idea of correlating multiple events together, you can do this in a single pass without a join e.g.  index=abc sourcetype=foo host=hostabc | eval boot_time=case(searchmatch("Linux version"),_time) | stats latest(_time) latest(boot_time) by host | rename latest(*) -> * | convert timeformat="%F %T" ctime(_time) as Latest_Event_Time ctime(btoot_time) as Boot_Time | eval delta=_time-boot_time, UP_Time = tostring(delta,"duration") | fields host Boot_Time Latest_Event_Time UP_Time   But the more practical problem you'll run into is the unbounded nature of how far in the past boot time can be... thus requiring this search to become almost an All Time search which doesn't scale well at all.  If you can add data sources... instead of relying just on this log, you could have a scripted input that captures the output of `uptime` on a regular basis.  But if not, another option may be to maintain a lookup containing the last boot time of a host, and pull that data in at search time instead... that way your search for looking at the latest events can be a much smaller window. Doing this off the top of my head, assuming a KVStore host_boots keyed by host, something like: index=abc sourcetype=foo host=hostabc | eval boot_time=case(searchmatch("Linux version"),_time) | stats latest(_time) latest(boot_time) by host | rename latest(*) -> * | lookup host_boots host OUTPUT boot_time AS last_boot | eval boot_time=coalesce(boot_time,last_boot) | fields - last_boot | outputlookup append=t key_field=host host_boots | convert timeformat="%F %T" ctime(_time) as Latest_Event_Time ctime(btoot_time) as Boot_Time | eval delta=_time-boot_time, UP_Time = tostring(delta,"duration") | fields host Boot_Time Latest_Event_Time UP_Time The question then becomes if you pull back this lookup for unseen hosts or not... and or if updating in this way makes sense (since the _time would get updated as frequently as the boot_time field...) and some other nuances... ... View more

So you might need to upgrade MLTK to 5.3.3, or follow the workaround to manually update conf files as listed in the Known issues under 5.3.1: https://docs.splunk.com/Documentation/MLApp/5.3.1/User/Knownissues ... View more

As it comes out of the box, the Splunk Add-on for Cisco ESA has no UI components.... If it was there'd be a default/data/ui folder which is missing... and the app.conf even states this isn't a visible app:   $ tar tzvf splunk-add-on-for-cisco-esa_160.tgz drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/ drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/LICENSES/ -rw-r--r-- 1001/121 85947 2022-07-25 01:18 Splunk_TA_cisco-esa/LICENSES/LicenseRef-Splunk-8-2021.txt -rw-r--r-- 1001/121 165 2022-07-25 01:18 Splunk_TA_cisco-esa/README.txt -rw-r--r-- 1001/121 1916 2022-07-25 01:18 Splunk_TA_cisco-esa/THIRDPARTY -rw-r--r-- 1001/121 11 2022-07-25 01:18 Splunk_TA_cisco-esa/VERSION -rw-r--r-- 1001/121 1551 2022-07-25 01:18 Splunk_TA_cisco-esa/app.manifest drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/default/ -rw-r--r-- 1001/121 473 2022-07-25 01:18 Splunk_TA_cisco-esa/default/app.conf -rw-r--r-- 1001/121 4770 2022-07-25 01:18 Splunk_TA_cisco-esa/default/eventtypes.conf -rw-r--r-- 1001/121 24749 2022-07-25 01:18 Splunk_TA_cisco-esa/default/props.conf -rw-r--r-- 1001/121 1208 2022-07-25 01:18 Splunk_TA_cisco-esa/default/tags.conf -rw-r--r-- 1001/121 50633 2022-07-25 01:18 Splunk_TA_cisco-esa/default/transforms.conf drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/ -rw-r--r-- 1001/121 85 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_authentication_action_lookup.csv -rw-r--r-- 1001/121 617 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_email_action_lookup.csv -rw-r--r-- 1001/121 920 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_proxy_status_action_lookup.csv -rw-r--r-- 1001/121 309 2022-07-25 01:18 Splunk_TA_cisco-esa/lookups/cisco_esa_vendor_info_lookup_160.csv drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/metadata/ -rw-r--r-- 1001/121 105 2022-07-25 01:18 Splunk_TA_cisco-esa/metadata/default.meta drwxr-xr-x 1001/121 0 2022-07-25 01:18 Splunk_TA_cisco-esa/static/ -rw-r--r-- 1001/121 3348 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIcon.png -rw-r--r-- 1001/121 3348 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIconAlt.png -rw-r--r-- 1001/121 6738 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIconAlt_2x.png -rw-r--r-- 1001/121 6738 2022-07-25 01:18 Splunk_TA_cisco-esa/static/appIcon_2x.png $ tar xOzvf splunk-add-on-for-cisco-esa_160.tgz Splunk_TA_cisco-esa/default/app.conf | grep visible Splunk_TA_cisco-esa/default/app.conf is_visible = false   (Lots of addons don't ship UI pieces, especially if they're only doing non-ui related things like setting up props and transforms, or lookup enrichments) Without knowing your stack or this app in depth, I suspect what is actually happening is that someone set the app to visible in your environment (which isn't needed), and more likely than not some other app is globally exporting its nav default.xml with a default view set that isn't being shared globally, thus when you open the app, you try to get to a view that isn't available and thus 404. (Using Settings > User Interface > Navigation Menus you can see if there's a nav bar visible in this app's context from a different app) Or there could be other quirks... but this add on has no UI components out of the box... ... View more

When you use tokens in dashboards they don't behave like variables, they behave more like #define macros in C... the literal value gets dropped into place where used prior to evaluation. (Ok this is an oversimplification, but when they're in your search strings... ) In the case that when your token has the value ALL/* then your where clause reads:  | where ENT_CallType=if(ALL/* =="*","*",ltrim(ALL/*,"VQ_")) To parse that statement in words where the field ENT_CallType has a value equal to ... if ( value of the field ALL divided by multiplied by ... And we've stopped making sense, thus we call it a syntax error when we reach the * coming from your token value. With the value VQ_abc_efg you don't wind up with the same syntax problem, but you do wind up with something that is obviously not what you were intending:  | where ENT_CallType=if(VQ_abc_efg =="*","*",ltrim(VQ_abc_efg ,"VQ_")) Again turning this into words, I'm looking in my results for a field called ENT_CallType, and seeing if it's equal to * when the field named VQ_abc_efg has the value *, otherwise the value of the field VQ_abc_efg removing VQ_ from the front of it...  I suspect you want to read up on Syntax to consume tokens , in particular $tokenname|s$ where your token value gets wrapped in double quotes before it gets inserted (and also helps with escaping quotations within the token value too). There are a few other hints around tokens there too. ... View more

I'd want to try it to be certain, but it sounds like it could be a job for a lookup... with WILDCARD match type defined for some columns: https://community.splunk.com/t5/Splunk-Search/Can-we-use-wildcard-characters-in-a-lookup-table/m-p/94513# ... View more

Typically, PaloAlto logs aren’t ingested as pan:traffic directly, but rather as pan:log (or older as pan_log) this gets changed into pan:traffic (and pan:other-log-types) during the transforms step assuming you have the Pan-TA: https://github.com/PaloAltoNetworks/Splunk-Apps/blob/develop/Splunk_TA_paloalto/default/props.conf So you likely need [pan:log] or [pan_log] in your props instead of [pan:traffic] depending on what your inputs look like on your forwarders Secondly you mention this is on your indexers. Are your PAN logs being ingested by Universal Forwarders or Heavy Forwarders? If they are Heavy Forwarders, or you are sending through  intermediate Heavy Forwarders, then parsing is already complete by the time you reach your indexers, and your props and transforms need to be on a different system (the first HF in the path from your syslog servers to your indexers) Hope this helps ... View more