We recently upgraded to from 7.1.2 to 8.0.3 on on-prem Splunk Enterprise. A previously working saved search is no longer returning the correct results. | transaction session_id maxspan=30s Looking into it looks like the transaction command is no longer closing connections when the maxspan (30s) value is hit. This leaves all transactions open and then the search ends when it hits the default of 5000. I need to create transactions out of 650000 entries (two or three lines each), so needless to say this search no longer functions. I can confirm this behavior, by: | stats count by closed_txn shows all the transactions returned as closed_txn=0 adding maxopentxn=5500 to the transaction command causes the number of returned results to go from 5000 to 5500 adding maxevents=2 only closes some of the events closed_txn , eventcount , count 0 1 1041 0 2 4458 1 2 1654 Transactions are supposed to close when: The ' closed_txn ' field is set to ' 1 ' if one of the following conditions is met: maxevents , maxpause , maxspan , startswith . https://docs.splunk.com/Documentation/Splunk/8.0.3/SearchReference/Transaction -> Memory control options -> keepevicted ... View more

After upgrading the to 6.4, Splunk web would no longer start: Starting splunk server daemon (splunkd) ... Done [ OK ] Waiting for web server at https://127.0.0.1:8443 to be available.... WARNING: web interface does not seem to be available! I thought it was an issue with the web.conf, so I ripped that out and it still would not start. I then removed the SSL setting in the server.conf and the server started normally. I re-added the web.conf and it again restarted fine. It is a Comodo cert (with their weird chain). The certificate works find in the web interface. We use the same for both. I though maybe it wanted a password on the key file, so I added one, that did not help. Looking at splunkd.log I see: 05-17-2016 14:06:42.214 -0400 ERROR X509 - /opt/splunk/etc/auth/MYSERVER-01.key: unable to read X509 certificate file 05-17-2016 14:06:45.156 -0400 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/MYSERVER-01.key errno=185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch. When the certificate is configured with the web interface, it passes all the verification checks. ... View more

We are ingesting Aruba CearPass logs. The ClearPass Appliances send their syslog to a syslog server that writes the logs to disk and then reads those log lines into Splunk. The log lines look like: <143>2016-03-07 18:04:57,504 yyy.yyy.yyy.yyy CPPM_Dashboard_Summary 35249531 1 0 session_id=R022d8e6d-04-56de08db,req_source=RADIUS,user_name=user@local.domain,service_name=WIRELESS_LOCAL,alerts_present=0,nas_ip=xxx.xxx.xxx.xxx,nas_port=0,conn_status=Unknown,login_status=ACCEPT,error_code=0,mac_address=abcdef123456,timestamp=2016-03-07 18:03:55-05,write_timestamp=2016-03-07 18:03:56.93952-05 The last key value pair is "write_timestamp=2016-03-07 18:03:56.93952-05", but Splunk records it as "write_timestamp=2016-03-07" this affects other field as well: <143>2016-03-07 18:09:52,982 yyy.yyy.yyy.yyy CPPM_Proc_Stats 390387 1 0 id=4529414,process_id=17,cpu_usage=0,res_mem_usage=3888,virt_mem_usage=188044,timestamp=2016-03-07 18:08:07.536779-05 "timestamp=2016-03-07 18:08:07.536779-05" becomes "timestamp=2016-03-07" ... View more

This is on a forwarder. We have two forwarders receiving syslog from some appliances. The forwarders write the syslog to disk and then the Splunk forwarder monitors for the files. The input stanza is: [monitor:///opt/inboundlogs/10.10.10.10/*_syslog.log] host = 10.10.10.10 disabled = false source = $HOSTNAME 10.10.10.10 sourcetype = vm_app index = app_foo The file name is /opt/inboundlogs/10.10.10.10/YYYY-MM-DD-HH_10.10.10.10_syslog.log and now our Splunk server is getting full of sourcetypes. I have set HOSTNAME in splunk-launch.conf and can see that Splunk sees it: /opt/splunkforwarder/bin/splunk envvars HOSTNAME=FORWARDER-01 ; export HOSTNAME ; PATH=/opt/splunkforwarder/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/splunk/bin ; export PATH ; SPLUNK_HOME=/opt/splunkforwarder ; export SPLUNK_HOME ; SPLUNK_DB=/opt/splunkforwarder/var/lib/splunk ; export SPLUNK_DB ; SPLUNK_SERVER_NAME=SplunkForwarder ; export SPLUNK_SERVER_NAME ; SPLUNK_WEB_NAME=splunkweb ; export SPLUNK_WEB_NAME ; LD_LIBRARY_PATH=/opt/splunkforwarder/lib ; export LD_LIBRARY_PATH ; OPENSSL_CONF=/opt/splunkforwarder/openssl/openssl.cnf ; export OPENSSL_CONF ; LDAPCONF=/opt/splunkforwarder/etc/openldap/ldap.conf ; export LDAPCONF And updated the stanza to: [monitor:///opt/inboundlogs/10.10.10.10/*_syslog.log] host = 10.10.10.10 disabled = false source = $HOSTNAME 10.10.10.10 sourcetype = vm_app index = app_foo But on the Splunk indexer, the source is "$HOSTNAME 10.10.10.10" and not "FORWARDER-01 10.10.10.10". I am planning on rolling this config into a Spunk App for easy management of the multiple forwarders receiving and forwarding on this syslog data, so I need the app/default/inputs.conf to be general and then I can set server specific settings with environment variables in the splunk-launch.conf. Using the app/local/inputs.conf to set this would suck, as there are currently eight incoming syslog streams, and more to come. ... View more