Regex101.com as shown in the links they provided will be your godsend when it comes to creating and testing regexes. ... View more

Try this on for size: index=newIndex host="1.12.123.4*" "Field"="abcd"| stats count as totalcount | where totalcount >= 10 | join type=outer host [search index=newIndex host="1.12.123.4*" "Field"="qwer"| stats count as totalcount | where totalcount >= 20 ] ... View more

I'd do it a little more like this personally: rex field=_raw "(?:.+at\s+)(?<time>.*(AM|PM))" ... View more

It only took me like three edits to realize that I wasn't messing up my typing but that I needed to put the named capture group in a code box... Jeezz.... ... View more

Because the Data Models are what delivers the data to the app. It doesn't pull logs directly for most of the graphs etc. And if the DataModels are empty, or failed to finish, or got messed up, then the app won't display any data. Of course this is assuming that PS set everything up correctly, and that the indexing is being done properly etc. ... View more

index=...| search MESSAGE="CommonAsyncGETController.execute() : scope :S01234"| rex field=MESSAGE "(?:.+scope\s\:)(?P<Scope>\S+)" | Table MESSAGE Scope At least, assuming that your original search is correct, that should give you a whole new field at search time called Scope, and S01234 will be the value. That being said, your search isn't going to find anything where the scope field would have a different value other than S01234... But that's a different question... ... View more

Here's the matrix of what can go with what: https://docs.splunk.com/Documentation/Splunk/7.2.3/Deploy/Manageyourdeployment And the Hardware Specs... https://docs.splunk.com/Documentation/Splunk/7.2.3/Capacity/Referencehardware That being said...I'd try not to combine if you don't have to. ... View more

DO NOT install Splunk to the default windows File Path i.e. C:\Program Files\Splunk . You'll find that Splunk can't create certain temp files (hashed search temp directories and files to be specific) due to the fact that the file path violates the Windows 260 character file path limitation. What'll end up happening is that Splunk will sucking up all the RAM available until SplunkD finally completely crashes, since it has nowhere to store stuff temporarily. Amazingly, the longest file path I've ever seen Splunk try to create unsuccessfully is 264 characters. Really? 4 Characters? Although I suppose that it could be longer or shorter depending on the name of the Search head referenced in the path. You Splunk coder guys couldn't have found somewhere to shorten up this file path or restrict it to less than the maximum number of characters allowed by the OS? For example: *C:\Program Files\Splunk\var\run\splunk\dispatch\remote_SearchHeadName_scheduler__admin_c3BsdW5rX2FwcF93aW5kb3dzX2luZnJhc3RydWN0dXJl__RMD5e93ff07c552f3ee0_at_1477516800_3187_F5AAE4E2-7A34-4327-8CDA-83913FB48502\index_buckets.csv.647C07D6-2813-4D98-AD2E-ED1FCACEB554.tmp* I mean, you've already got multiple Hashes going on here, can't you just remove the index_buckets.csv. part and save like 17 characters? Just goes back to the whole issue of Windows Installs being treated like the second class installs. Yes, supposedly this restriction is removable in Windows 10, and hopefully that means the same is true for Server 2016, but I haven't found any documentation to definitively state that one way or another. Also given that there's a requirement to enable it via either registry key , or Group Policy , and that there are other caveats, I don't fully trust Windows to support this functionality, nor Splunk's ability to access it reliably. Trust me, you're better off avoiding the whole issue entirely, and just installing to C:\Splunk or D:\Splunk or whatever drive letter you'd prefer. Better yet, Go NIX. ... View more

Try this: source=/home///log/*| rex field=source "(?P[^aA-zZ].[0-9].log)" Regardless, the command you need is the rex field=fieldname"regex_goes_here" part to get the regex working within the search. https://docs.splunk.com/Documentation/Splunk/7.2.3/SearchReference/Rex ... View more

I'd bet a dollar that the deployment server is Windows based. Windows based deployers totally foul up the permissions of the TA_nix when they push it. ... View more

Rebuild the data models. ... View more

Additionally, the the API expiration alert will ALWAYS fire as it's currently written. The search is written to get 1 event (per the "dedup customer_name" portion of the search). Eval is done to determine the number of days left before the API expires, but that's not taken into account by the alert The alert is fired whenever the number of results of the search are less than 30. The number of results of the search are ALWAYS 1 which is always less than 30. Therefore the alert always fires. Add this: | search "Days Left"<30 to the end of the API Expiration Alert search and change the alert to fire whenever the results are GREATER than 0, and you'll achieve the desired effect of firing the alert whenever there's less than 30 days before the API expires. ... View more

Also just for the record, moving the default installation to a shorter path also fixed our problems with our ES search head as well, and now things are much better! We also changed our ES search head to be installed on Linux, and got better performance in general, but in theory if you had ES running on a Windows based search head, changing the install path on that as well should alleviate any problems you'd have on the search head with temp files, and changing the install path on the indexers I can definitively say alleviates the errors that were occurring on the indexers. ... View more

Circling back to close this up... finally... Nope, Turns out the best solution for us was to reinstall Splunk, and move the necessary Conf files from the original location to the new one, as was jkat's original solution. ... View more

Just for the record...I didn't create the searches that are causing this problem. I finally figured out where they're coming from. It's the pre-defined searches (aka "Guided Setup") that the Splunk App for Windows Infrastructure runs to verify that the proper data is flowing into Splunk, and to pre determine what panels to setup in the App. Also, Symbolic Linking it wouldn't alleviate the problem, unless you could tell Splunk to use the sym link instead of the default path. The problem isn't that there's no space, or that the space can't be accessed. I could in theory create an H:\ and then add 257 characters again for a grand total of 500 characters, but windows won't let you work past 260 in the NAME of the path, so unless splunk lets me change the file path to the dispatch directory, windows won't allow it. ... View more

sourcetype=wineventlog =22 characters. You're not going to get anywhere with a 20 character search, so I agree that's a bad option. I could move the Splunk install location, but even that's not a great option, and would take some time to rip it out, clean it up, and put it all back together, then repeat two more times to get all three indexers back up. This is definitely something that's in need of a bug/enhancement request. Even better than the request to specify the number of chars in hashes, would be to allow us to specify the location where the hashed directories are created. Then I could map a drive to wherever I wanted, and just use the letter to specify where they go. For example if there was a conf that included: [hashConfig] hashPath = H: then I could map H to C:\Program Files\Splunk\var\run\splunk\dispatch\ And then the files would be in the exact same location, but would be accessible because the file path for H:\ is only 3 characters. ... View more

I agree that will be lovely... In the future. Assuming they don't back out again, like they did for Server 2k12. ... View more

If you're on a paid version you can call Splunk and have them reset your license, (as long as you're still implementing, and configuring, and haven't abused their graciousness...) ... View more

Here's a good place to start: http://blogs.splunk.com/2012/09/21/the-splunk-app-for-active-directory-and-how-i-tamed-the-security-log/ ... View more

Make Sure that the VMware Inventory and VMware Performance Data Models are being accelerated, otherwise those Graphs don't get populated., ... View more

Glad you found the answer! ... View more