We've recently seen a significant spike in memory utilization on our search heads ... Looking at the files opened by mongod I'm seeing info like this: -rw------- 1 root root 16777216 Dec 7 16:37 s_SA-AccyUyi@longstring@.0 -rw------- 1 root root 33554432 Sep 15 20:58 s_SA-AccyUyi@longstring@.1 -rw------- 1 root root 536608768 Nov 16 02:36 s_SA-AccyUyi@longstring@.10 -rw------- 1 root root 536608768 Sep 19 23:28 s_SA-AccyUyi@longstring@.11 -rw------- 1 root root 536608768 Dec 7 16:38 s_SA-AccyUyi@longstring@.12 -rw------- 1 root root 67108864 Sep 15 20:58 s_SA-AccyUyi@longstring@.2 -rw------- 1 root root 134217728 Dec 7 16:29 s_SA-AccyUyi@longstring@.3 -rw------- 1 root root 268435456 Dec 7 16:38 s_SA-AccyUyi@longstring@.4 -rw------- 1 root root 536608768 Sep 19 23:28 s_SA-AccyUyi@longstring@.5 -rw------- 1 root root 536608768 Dec 7 16:38 s_SA-AccyUyi@longstring@.6 -rw------- 1 root root 536608768 Dec 7 16:38 s_SA-AccyUyi@longstring@.7 -rw------- 1 root root 536608768 Sep 19 23:32 s_SA-AccyUyi@longstring@.8 -rw------- 1 root root 536608768 Sep 19 23:31 s_SA-AccyUyi@longstring@.9  Any idea why there are so many versions of some of these? ... shouldn't there typically only be a ".0" and a ".ns" file for each collection?   Thank you   ... View more

Having trouble finding documentation on these two parameters. But, say column 3 is an epoch timestamp, would input_timestamp_format be a single 's'? ie: input_timestamp_format = s Thank you. ... View more

I've been working on automating our UF upgrade process and have found what appears to be an issue with a deprecated key, sslKeysfilePassword ... When I upgrade an old 6.1 or 6.2 host beyond Splunk 6.5, I've found that while the UF can still maintain forwarding over SSL to our indexers, they can no longer handshake with our deployment server. Spending most of my week on this, I've come across a workaround where, prior to performing the upgrade ... ( stopping splunk; tar -zxf blah ), if I remove the deprecated key "sslKeysfilePassword" from etc/system/local/server.conf ... the handshake problem is no longer an issue. The odd thing here is that, this is the only thing that had to be changed to rectify the issue, but my understanding of a deprecated object is that it would just be ignored. It doesn't appear to be the case in this instance. So, this isn't really a question perse, but has anyone ever run up against this before? ... View more

The raw data looks like: ... blah, blah, blah ... "detail-type": "GuardDuty Finding", "time": "2019-03-14T14:40:39Z"} On our Heavy Forwarder I've setup in Splunk_TA_aws/local [aws:kinesis] TIME_PREFIX = \s+\"time\":\s+\" TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ MAX_TIMESTAMP_LOOKAHEAD = 40960 The regex works on the standalone Splunk instance I have on my laptop, it works on regex101.com But when the data is indexed, the time Splunk indexes on my cluster is: 3/14/19 2:46:29.090 PM Any clues as to what might be happening here? ... View more

We've recently added 50% more indexers. After rebalancing the cluster, we're finding that we still have a gap on our hot/warm storage where, the new indexers have maybe 2 or 3 warm buckets, and the older ones have maybe 20 or 30. The splunk rebalance cluster-data command has definitely evened the total bucket counts between the indexers, but is there a way to balance only the warm buckets? (I believe the answer to that is 'no'). ... View more

I'm attempting to update our certs between our universal forwarders (UF) and indexers in our test environment. I believe I have the certs properly generated and in place. But when the UF attempts to forward, we see this error: 10-19-2018 08:13:14.661 -0600 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'. 10-19-2018 14:17:44.863 +0000 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'. 10-19-2018 14:17:44.863 +0000 ERROR TcpInputProc - Error encountered for connection from src=nn.nn.nn.nn:38438. error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher This leads me to believe that the cipherSuite needs to be updated ... indexer server.conf - ( Splunk 7.1.3 ] [sslConfig] sslVersions = tls1.2 sslVersionsForClient = tls1.2 cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 ( etc/system/local/inputs.conf under [SSL] ) cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH UF - Splunk 6.6.4 - etc/system/default/server.conf [sslConfig] cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 etc/system/default/outputs.conf [tcpout] sslVersions = tls1.2 cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256 I've been using this link to generate and set up the new forwarding certs. https://wiki.splunk.com/images/f/fb/SplunkTrustApril-SSLipperySlopeRevisited.pdf ... View more

As I go through the manual process of trying to migrate queries from dbConnect v1.x to dbConnect 3.1.3, I'm having issues with the Edit Input panel. I follow the steps. Choose a valid connection - check Browse structure and type SQL .... - check Pick a rising column and set the checkpoint value - check Update SQL to accept the checkpoint value and make sure it works. This is where I run into a problem.The second I start typing "WHERE TIME_STAMP > ? .... The Rising Column dropdown completely empties out and the query returns: com.microsoft.sqlserver.jdbc.SQLServerException: The value is not set for the parameter number 1. ( No results found ) This makes me unable to save the query and actually set a Rising Column and a Checkpoint Value If I execute the same query using EVENT_TIME, things work ... but, both EVENT_TIME and TIME_STAMP are 'bigint' objects, and they both show up in the batch query results, so the question would be, why is TIME_STAMP an invalid field to use as a rising column in dbConnect 3 but works perfectly in dbConnect 1? The target database is MS SQL Server ... View more