Almost 2023 in Splunk 9.x and it's still an issue... ... View more

Oops, I thought I had replied back to this thread with the final solution.  To your point, it was an issue with the ip allow list on the ServiceNow side.  Once we added the Splunk Cloud search head IPs to the SNOW allow list we were able to connect as expected. Thanks! ... View more

Yea.  I feel like it must have something to do with how the UI handles these.  It seems like it's using regex and it's a bit overzealous on that regex.  Configuring the same delimited (as delimited) from a back-end perspective works fine without issues... ... View more

The point here being, that's not my regex.  That was generated by the Splunk UI when I tried to create a field extract using 'delimiting with pipe'...  The only reason I have that regex in hand is because the error message included it... ... View more

No doubt the regex can be improved significantly as you demonstrated.  I guess my challenge is, how do I tell my users that the OOB delimited option just doesn't work and that they now have to go learn regex to extract their fields? At the end of the day, I see 3 possibilities... I'm doing something wrong... The default limits are just too low and I should increase them (to what?)... Splunk's delimited parsing UI just generates really inefficient regex..   ... View more

I think the 403 is unrelated and probably related to the specific test that was run. I have attempted to configure the ServiceNow addon from both Splunk Cloud as well as a local instance of Splunk Enterprise running on my workstation.  Using the same info (client id/secret) I can establish a connection with my local Splunk instance, but experience an error when attempting to do the same in Splunk Cloud. Splunk Support pulled the splunk_ta_snow_main.log at my request from the search head in question and the below error occurs every time we try to complete the OAuth exchange... 2022-01-24 13:03:04,742 INFO pid=12591 tid=MainThread file=splunk_ta_snow_rh_oauth.py:getProxyDetails:121 | Proxy is not enabled 2022-01-24 13:03:04,768 ERROR pid=12591 tid=MainThread file=splunk_ta_snow_rh_oauth.py:handleEdit:106 | Error occurred while getting access token using auth code Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/splunk_ta_snow_rh_oauth.py", line 95, in handleEdit content = json.loads(content) File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)   Given this same exchange works from a local Splunk Enterprise instance, it tells me that this is not an issue on the ServiceNow side... I've increased the logging level and requested another set of logs.  Hopefully it will give a little more insight into what is going on... ... View more

We ran into this same issue unfortunately.  A previous administrator set it up using Generic S3.  Over time the performance got so bad we tried using Incremental S3 and ran into this issue.  Due to the history and the need to 'just make it work', we went the route of modifying the app to include the org id in the path. The caveat here is that this means you can't use the CloudTrail Incremental S3 input for anything else unless it contains the same org id structure.  It also means we're running the app on an on-prem forwarder and not within Splunk Cloud... In our case our plan is to move to streaming logs through HEC via Amazon Kinesis, so the workaround is temporary.  Otherwise I would have not agreed to hacking up the add-on in the first place... File Splunk_TA_aws/bin/splunk_ta_aws/modinputs/incremental_s3/cloudtrail_logs.py def _enumerate_partitions(self, s3, bucket): partitions = list() #prefix = self._prefix + 'AWSLogs/' prefix = self._prefix + 'AWSLogs/o-1234567890/'   ... View more

Hopefully it helps.  In my situation the AWS Add-on is running behind a proxy that essentially 'middle  mans' the connection so it can be scanned by a DLP product.  The certs are signed by our internal signer, so out of the box this Add-on will not work.  Adding our root internal signer works like a champ... Good luck! ... View more

One option would be to add your internal signer to the ca certs list python is using to validate the certificate.  It still requires you modify the AddOn, but it's at least maintaining the security of the connection... The certifi module contains the ca certificates file and can be found at SPLUNK_HOME/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem.  You can toss your internal signer at the end of the file... ... View more

Many SAML providers can also provide 2FA services.  All of the authentication and 2FA is handled by the provider and Splunk doesn't have to know anything about it.  From Splunk's point of view, the provider has done whatever it needs to authenticate/authorize and just asserts the identity. ... View more

All communication to/from S3 is going to be done over SSL, so the encryption is going to be handled for you.  As far as encryption at rest, AWS KMS is going to handle that encryption/decryption on the AWS S3 side to ensure the file is encrypted at rest. ... View more