So I'm trying to simulate enabling SSL from all aspects of Splunk and I can't get the forwarder to talk to the indexer at all. I've followed along with both .conf presentations regarding SSL and the Splunk docs > securing Splunk and I can't get it to work. I can get Splunk Web to work as https using my signed certs. The indexers open the port with my signed certs and enable to the port to be ssl; however, forwarders don't work at all. I get this error: 02-11-2016 15:45:40.789 +0000 ERROR TcpOutputFd - Connection to host=xxx.xx.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed Does anyone have any troubleshooting techniques that can help solve this? Seems like all the questions on here are unanswered and really would like to get over this issue. Here is my inputs.conf on my local indexer [SSL] rootCA = $SPLUNK_HOME/etc/apps/idx_ssl/local/ca-cert.pem serverCert = $SPLUNK_HOME/etc/apps/idx_ssl/local/server.pem password = password # sslVersions = tls # requireClientCert = true [splunktcp-ssl:9997] compressed = true Here is my outputs.conf on my forwarder [tcpout] defaultGroup = splunkssl [tcpout:splunkssl] server = xxx.xx.x.x:9997 sslCertPath = $SPLUNK_HOME/etc/apps/fw_ssl/local/server.pem sslRootCAPath = $SPLUNK_HOME/etc/apps/fw_ssl/local/ca-cert.pem sslPassword = password # sslVerifyServerCert = true # sslCommonNameToCheck = Splunk Things that are commented out where also uncommented and same results. Just thought I would include all of what I've tried. All these values are local development, nothing prod. ... View more

Hello fellow Splunkers, Have any of you been able to install Splunk Universal Forwarder on Solaris using the PKG file? I'm trying to script it so that it is installed silently without any interactions. Has anyone been able to achieve this? Might be a basic one, but can't seem to find that answer anywhere. ... View more

I'm looking to get information on what is the best way to make indexes for data. Background: Setting up a clustered environment with both cluster indexers (replication factor 3) and clustered (distributed? Still confused on clustered search head vs distributed search) search heads. Our current approach is to put an "application" in its own index collecting anything the Universal Forwarder will send us including log data and basic OS data. Question: Our new approach is to create an 'OS' index to handle all UF OS stats from all servers and use tags to help with search, and to create multiple indexes per app to make an index have common data. We have around 170 applications we want to start monitoring as they are our class 1 and class 2 apps so that would be 3 x 170 = 510 indexes. Is this a horrid approach? Example: Application X - Index OS - Index X - Index X_Critical (for when we need to ramp up interval time for troubleshooting real time and would be cleaned out after ... View more

Long story short. We use DB Connect to pull data. Recently packaged an app for testing purposes to put on another instance, but realized DB Connect information wasn't being included in the app when packaged since the Name Input > App was set to Splunk DB Connect. Changing that seems to have fixed my packaging issue and works great. Here is the real question. We recently decided to copy the stanza in inputs.conf and manually create 6 more since db table was the only thing changing. Rising column is set and using that to track changes. When I added the new stanzas, the data was indexing properly. I looked in the inputs and noticed that {app folder}/local/inputs.conf looked great and nothing was changed, but noticed that {db connect 2.0}/local/inputs.conf was copying over the [mi_input] name and placing the tail_rising_column_checkpoint_value=value here and a disable=1 - thats it. Is this normal? And/or something I need to worry about? Could some sort of permissions regarding the connection or something be causing this? Thanks ... View more