TO remediate this you can follow the below steps which also worked for me. Enabled the boot-start with splunk user. Check the directory permission which is set to splunk:splunk Exit the login session and you will find that Plunk again stopped. This means that enable boot-start has not created systemd service. Now manually create the splunkd.service under systemd/system. Unit file used is below Set the parameter "RemainAfterExit=yes" and reloaded the daemon. Restarted the Splunkd with systemctl. Exit the login session again and observer that Splunk service which should not be stopped ans should resolve the issue. Navigate to:- /etc/systemd/system/splunkd.service [Unit] Description=Splunk Enterprise 6.5.0 After=network.target Wants=network.target [Service] Type=forking RemainAfterExit=False User=splunk Group=splunk LimitNOFILE=65536 ExecStart=/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt ExecStop=/opt/splunk/bin/splunk stop PIDFile=/opt/splunk/var/run/splunk/splunkd.pid [Install] WantedBy=multi-user.target If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ... Alias=splunk.service If you still see the Splunk is crashing with the below message from /var/log/messages use the below parameter: May 18 21:30:56 pplsplunkapph25 abrt-server[6936]: Executable '/opt/splunk/bin/splunkd' doesn't belong to any package and ProcessUnpackaged is set to 'no' To solve this limitation, you need to edit the file sudo vim /etc/abrt/abrt-action-save-package-data.conf Then change the parameter ProcessUnpackaged to yes Process crashes in executables which do not belong to any package? ProcessUnpackaged = yes If still Splunk crashes, please raise a ticket with your internal admin team. REF DOC:- https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/RunSplunkassystemdservice https://answers.splunk.com/answers/59662/is-there-a-systemd-unit-file-for-splunk.html ... View more

Additional to that if you see the below ERROR as well you can also increase the value of maxConcurrentOptimizes in indexes.conf for a particular index which is affected or you can set globally to all the indexes depends on your requirement. ERROR: 04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child_39286__SplunkOptimize) optimize finished: failed, see rc for more details, dir=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hot_v1_268, rc=-12 (unsigned 244), errno=0 04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child_39286__SplunkOptimize) merge failed for path=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hot_v1_268 rc=-12 wrc=-12 errno=0 file=dontknow hint=tsval_id is UINT_MAX in _merge_all_postings] Example: In indexes.conf For particular index: [_internal] maxConcurrentOptimizes = < desired value > For all the indexes: [default] maxConcurrentOptimizes = < desigered value > maxConcurrentOptimizes = < nonnegative integer > The number of concurrent optimize processes that can run against the hot DB. This number should be increased if: There are always many small tsidx files in the hot DB. After rolling, there are many tsidx files in warm or cold DB. Must restart splunkd after changing this parameter; index reload will not suffice. Highest legal value is 4294967295 Defaults to 6 ... View more

Hi, The meaning of this message is that the indexers are busy, and the queues full. Therefore the internal splunk logs (like audit) are disabled in order to dedicate all the performance to the indexing. please check - apparently your Splunk instance is forwarding to itself. Check on the indexer: Is a receiving port set? [okay] Is the indexer forwarding? Where? If it is forwarding to itself, then that's the problem! You can find both of these settings in the UI under Settings>>Forwarding and Receiving Or, you can find the receiving settings in inputs.conf and the forwarding settings in outputs.conf Please accept this answer, it this pointed you in the right direction. ... View more

Hi, Are you using Splunk Universal forwarder or Splunk Enterprise? If UF --> then there is no web GUI. The Splunk web is available only for Enterprise versions. It's the same whatever the machine you are using(VM/cloud/remote). Only thing is to check the communication from your localhost to VM. If you haven't change the port number, then the below should work. If you change, then use that port number. Check the web status and port number. splunk show web-port and splunk status splunkweb. access the Splunk Web GUI from the same server as --> http://test:8000 OR http://localhost:8000 access the Splunk Web GUI from the different/remote/VM server as --> http://test:8000 OR http://10.10.10.10:8000 ... View more

Check the connection between both the Windows servers via ping protocol. If both can communicate you should access the splunk URL. Use this URL from the remote machine to access "http://hostname of splunk server OR splunk server IP address:8000". For example: If Splunk is installed in the server with a name --> test --> ipaddress: 10.10.10.10 access the GUI from the same server as --> http://test:8000 OR http://localhost:8000 access the GUI from the different server as --> http://test:8000 OR http://10.10.10.10:8000 "http://localhost:8000" won't be working as you're trying to access from the different server. Localhost is used if you are trying to access the splunk GUI in the same server where splunk is used ... View more

RAFT is a Splunk Search Head Cluster Terminology" Raft distributed consensus". The process of Electing the search head captain dynamically is known as RAFT. Auto SH captain failover Elect new captain via RAFT, it is recorded in $SPLUNK_HOME$/var/log/splunk/_raft//log. This file contains: **Members register their list of artifacts, running jobs, alerts, and search load statistics to a new captain. **New captain enables its scheduler and executes fix-ups if needed. If it is corrupted and a crash occurs follow below doc to troubleshoot. https://docs.splunk.com/Documentation/Splunk/7.2.3/DistSearch/Handleraftissues sample log in the crash.log - "what(): error in raft entry file for node term Could not parse raft entry file. Search head clustering: Search head cluster member has a corrupted raft state. 2019-01-08 07:59:43.450 +0000 splunkd started (build a0c72a66db66) " ... View more

Its true that Not all Splunk cookies have the HttpOnly tag set Apply the below fix for default settings: web.conf to see if tools.sessions.httponly is set to true http://docs.splunk.com/Documentation/Splunk/7.0.3/Admin/Webconf server.conf to see if cookieAuthHttpOnly is set to true http://docs.splunk.com/Documentation/Splunk/7.0.3/Admin/Propsconf However, cval is cookie test that needs access from JS (can't be httpOnly) csrf_token uses double submit pattern in JS (can't be httpOnly) These are intentional cookie parameters and not security relevant issues. They basically outline and explain that cookies are not related to any session or authentication settings and are needed to access from JavaScript, so they cannot be ‘HttpOnly’. The httponly flag is a mechanism to disallow the use of these cookies from script elements; however, these cookies are used by scripting elements, so setting them as httponly would break the web interface functionality. ... View more