I'm going to take my digging as the answer here. In 7.2.3 and 7.2.4.2, I'm finding that this may be due to a difference in how Google implements the S3 API. So currently, GCS is not a viable option for S2. For reference-- https://www.zenko.io/blog/four-differences-google-amazon-s3-api/ https://github.com/kahing/goofys/issues/259#issuecomment-355713879 > This is because GCS's S3 implementation does not support S3 multipart uploads and instead reinvented another API (ugh). [...] This includes testing various configurations like: - remote.s3.headers.POST.Content-Type //to resolve multipart-upload error - remote.s3.use_delimiter //to see if the guidSplunk delimiter was the issue - use_batch_remote_rep_changes //see if it was related to a race condition with the CM making calls to the peers ... View more

I was thinking something similar. Reading through the Google documentation here: https://cloud.google.com/storage/docs/json_api/v1/how-tos/multipart-upload It talks about "multipart/related", not "multipart/form-data" Digging further, I'm finding that this may be due to a difference in how Google implements the S3 API. https://www.zenko.io/blog/four-differences-google-amazon-s3-api/ and referenced here, https://github.com/kahing/goofys/issues/259#issuecomment-355713879 This is because GCS's S3 implementation does not support S3 multipart uploads and instead reinvented another API (ugh). Coincidentally I've started working on a fix for this so stay tuned. ... View more

No dice. On version 7.2.4.2 the above error regarding a POST expecting multipart-upload persists. Also, the use of the remote.s3.header.x.x config triggers and immediate crash still. The crash log shows it's cachemanager that's choking on it. ... View more

This could be it since the error is related to the multipart-upload command. It would've been nice if the release notes could've been a bit more specific with that it was resolving. I'm going to try the upgrade path in a test environment and see if that resolves that error. I'll report part my findings. ... View more

We're currently running 7.2.3. I know there are some dashboards added to the MC in 7.2.4, but we're not quite there yet. Also, the indexes that we're testing do not use DMA. That was part of our test prep. ... View more

The setting use_batch_remote_rep_changes needs to be in server.conf and placed outside of any of the stanzas. If it's in one of the stanzas, it'll be flagged as an invalid key. So just put it at the top of the server.conf in etc/system/local/server.conf. At least that's where I put it and didn't receive any errors from btool ... View more

In Postman or Soap UI, you'll want to make sure that Authorization (Basic Auth) is set and the credentials are added. Postman - Authorization https://learning.getpostman.com/docs/postman/sending_api_requests/authorization/ Soap UI - How to Authenticate https://www.soapui.org/soap-and-wsdl/authenticating-soap-requests.html ... View more

Are you using port 8089? https://your-instance.com:8089/services/admin... The UI runs on port 8000 and is typically forwarded from 443. But the API Endpoints are over 8089, which is also the management port. ... View more

Just a thought-- but wouldn't the cacheman only be aware of buckets that have rolled all the way to Cold and been called back because of a search? ... View more

There are several ways that it could be approached, but one rough way would be as follows: | makeresults | eval path = "abc.com/?a=1&b=2 abc.com/?c=1 abc.com/?d=1&a=2" | makemv path | mvexpand path | rename COMMENT AS "Everything above generates sample event data; everything below is your solution" | makemv path delim="?" | eval params=mvindex(path,1) | makemv params delim="&" | mvexpand params | stats values(path) as Paths by params ... View more

My own experience with 7.1.2 at scale in a hybrid environment (mostly cloud, but a lot of forwarders on-prem and a handful of heavies) has seen a lot of memory issues and quirks in Indexer cluster. We're, thankfully moving to 7.2 in the next few days. YMMV, though, since obviously the impact would depend on your own install. And bonus, 7.2 has dark mode and a slew of other features. 7.2 Release Notes ... View more

I should also note, it's an input time config. So, from a UF perspective, it would do nothing. It'd need to be on the indexer side to have any effect on the parsing. At least that's my takeaway from other answers here and the spec page for props I just ran into that recently testing changes and I had to upload sample logs to validate that particular parameter was working as expected. ... View more

Unless it's a structured format like CSV, JSON, or XML, a UF would not be able to parse it. ... View more

@hnelsonit, Did you end up still using the setnull prop as well? Or did the your solution effectively only index the parsed field? ... View more

What I use to refresh or reload an app is a python script on the command line. It could obviously be tweaked to be automated using stored credentials, but for my purposes, I only use when developing to update when I make changes. Here's the script in it's entirety: import sys import splunklib.client as client import getpass splunkServer = raw_input('Enter the splunk server: ') splunkAdmin = raw_input('Enter Splunk Admin: ') splunkPassword = getpass.getpass('Enter Splunk Password: ') splunkDestApp = raw_input('Enter target App Context: ') if splunkDestApp: splunkService = client.connect(host=splunkServer, port=8089, username=splunkAdmin, password=splunkPassword, app=splunkDestApp) else: splunkService = client.connect(host=splunkServer, port=8089, username=splunkAdmin, password=splunkPassword) print('Refreshing app entity...') applications = splunkService.apps for app in applications: if splunkDestApp: if app.name == splunkDestApp: try: #app.refresh() app.reload() print(splunkDestApp+' has been refreshed') except EntityDeletedException: print('Application '+splunkDestApp+' does not exist.') The gist is, it prompts for the server and credentials and then which app is to be reloaded. I run it using: >splunk cmd python scriptname.py ... View more

Yes. In fact, it was immediately better. Once it went out with the deploy, the initial response, TTFB, dropped from 10+ seconds to between 1-4. In our UI cluster, we added the following to web.conf. [settings] ... max_view_cache_size = 2000 ... ... View more

As an update, I'm going to try max_view_cache_size in web.conf. We have a 1000+ dashboards (views) in the Search app. The test will be to see if that improves the performance. ... View more

Version 64.0.3282.186 (Official Build) (64-bit) Nothing is blocked. The waterfall is clean, so to speak, everything loads. No Job Inspector since this is the initial load of the Search app, i.e. no search has even run. Which is why, I can see that the issue is sitting on the server (search head) side. The time to first byte is how long the browser is waiting for Splunk (splunkweb process) to process the and sent back the response. So the question is, from a web framework point of view, what does Splunk do on the initial load. It would seem that there's zero caching of UI elements and it's generating everything each time. ... View more

I'm seeing similarly poor performance as well in the UI. The TTFB is very slow in the UI cluster. ~14s until First Byte. We have theory that it may be due to the sheer number of dashboards in the Search app versus say specific applications that have been setup in the UI. But it's been challenging to nail down what the splunkweb process is looking to load in the initial request before it send it's first byte. ... View more

@sdemoss - With the approach you took for Cisco Wireless telemetry, was it necessary to make one call to get a list of devices and then N number of calls per device returned where N is the number if metrics? We're in the process of doing something that sounds similar and I'm sizing up the effort, but reading through the API documentation makes me think that I'm going to need thousands of calls to gather the metrics. ... View more

We're seeing the same issue. When in the UI, the validation fails when loading the driver. The same workaround of going to the Settings and Reloading the drivers temporarily allowing the connection to be completed or a query setup. ... View more