It is caused by default out of box settings in the ES data model configurations. After a fresh installation or upgrade to 4.7.x , users need to adjust ES data model settings according to their envionrment and business needs. By default, all ES data models are configured to search across ALL indexes, which will result in extremely high memory utilization at the indexers. This is also documented in Splunk manual on http://docs.splunk.com/Documentation/ES/4.7.2/Install/Datamodels#Constrain_data_model_searches_to_specific_indexes it states: "You can constrain the indexes searched by a data model to improve performance. By default, data model acceleration searches search all indexes, which can lead to high memory consumption on indexers" audit.log can show inefficient searches are running across all indexes "((()))" e.g. search='summarize tstats=t action=probe id=DM_Splunk_SA__ CIM_Change_Analysis normid= [ search (index=* OR index=_) *((()) tag=change) audit.log. 09-08-2017 22:01:31.971 -0400 INFO AuditLogger - Audit:[timestamp=09-08-2017 22:01:31.971, user=splunk-system-user, action=search, info=granted , search_id='SummaryDirector_1504922491.13494', search='summarize tstats=t action=probe id=DM_Splunk_SA_CIM_Vulnerabilities normid= [ search (index=* OR index=_) *((()) tag=vulnerability tag=report) ... Good search should be without "((())" 09-10-2017 12:46:02.573 -0400 INFO AuditLogger - Audit:[timestamp=09-10-2017 12:46:02.573, user=splunk-system-user, action=search, info=granted , search_id='scheduler_nobody_U3BsdW5rX1NBX0NJTQRMD5cefc72a72dd5ee92_at_1505061960_7217', search='| summarize tstats=t override=partial manual_rebuilds=t max_time=3600 poll_buckets_until_maxtime=f id=DM_Splunk_SA_CIM_Vulnerabilities [ search (index=* OR index=*) (((index="whois" OR index="wineventlog")) tag=vulnerability tag=report) ... How to fix? In ES app, navigate to Configure -> CIM setup. Select any data model -> under the indexes tab, select indexes that are used by this particular datamodel. How to find the list of indexes searched by this data model? Run tstats search against data models by index e.g. | tstats count from datamodel=Malware by index | tstats count from datamodel=Authentication by index | tstats count from datamodel=Application_State by index | tstats count from datamodel=Network_Traffic by index 3.Save changes. 4.Navigate to Settings -> Data Models > rebuild data model. 5.Check in audit.log searches should like similar to this, without "((())" id=DM_Splunk_SA_CIM_Vulnerabilities [ search (index=* OR index=*) (((index="whois" OR index="wineventlog")) tag=vulnerability tag=report) 6.Repeat the same steps with all other ES data models. ... View more

As per error message, root cause of the issue is related to log.cfg file in located the /opt/splunk/etc/ directory. Checked in the splunkd.log file in $SPLUNK_HOME/var/log/splunk and observed the following message in it: WARN Logger - /opt/splunk/etc/log.cfg:11: Parse error at "TailingProcessork.... It was found that the category.TailingProcessor parameter name was not spelled correctly. After updating the parameter name in the log.cfg file, user was able to start Splunk Server successfully. ... View more

In Splunk 6.2.3 release stanza should be [WinEventLog:Security] instead of [WMI:WinEventLog:Security] For more for more details about the stanza settings in inputs.conf, please refer to Splunk Documentation: http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/MonitorWindowsdata ... View more