Thank you! Same issue here on Splunk 9.2.1 Splunk was NOT starting at boot-start (with init.d) but manually was starting correctly. After having commented the mentioned line is now properly booting with the VM (Oracle Linux). I am going to open a case to the support to inform them about it. ... View more

@dnavara please have a look on my explanation here: https://community.splunk.com/t5/Getting-Data-In/Why-has-the-index-process-paused-data-flow-How-to-handle-too/m-p/631226/highlight/true#M108187   ... View more

Thank you all, here are updated configurations adapted from the previous ones. NSG logs are written into an Azure Storage Account. Then a Splunk HF reads the logs from the Azure Storage account with "Splunk Add-on for Microsoft Cloud Services" and send back to the Indexers.   Configuration applied on the Splunk Heavy Forwarder (can be applied in an Indexer if you don't have an HF) hf_in_azure_nsg_app/default inputs.conf #Inputs is defined directly in the Splunk HF via WEB-UI with "Splunk Add-on for Microsoft Cloud Services" and can be found here /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local/inputs.conf props.conf #NOTE: Following set-up allow to extract only the flowTuples from the payload and set _time based on flowTuples epoch #First LINE_BREAKER apply, then SEDCMD-remove_not_epoch that keeps only flowTuples, then TRANSFORMS with INGEST_EVAL that overwrite _time #flowTuples data parsing is done at search time in the Search Head with separate app #The "source" field already contains the resourceId informations (subscriptionID, resourceGroupName, nsgName, macAddress) that can be extracted on the Search Head at search time #NOTE 2: LINE_BREAKER has been enhanced to avoid extracting events with macAddress containing first 10 numeric digits #TO BE DONE: Understand if SEDCMD- has some limit on very huge payload #TO BE DONE 2: In the INGEST_EVAL with a case statement if length is lower than 10 digits valorize now() as _time #References: #https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview #https://community.splunk.com/t5/Splunk-Search/How-do-I-import-Azure-NSG-LOGs/td-p/396018 #https://community.splunk.com/t5/Getting-Data-In/How-to-extract-an-event-timestamp-where-seconds-and-milliseconds/m-p/428837 [mscs:nsg:flow] MAX_TIMESTAMP_LOOKAHEAD = 10 LINE_BREAKER = (\")\d{10}\,|(\"\,\")\d{10}\, SHOULD_LINEMERGE = false SEDCMD-remove_not_epoch = s/\"\D.*$|\{|\}|\[|\]//g TRUNCATE = 50000000 TRANSFORMS-evalingest = nsg_eval_substr_time transforms.conf [nsg_eval_substr_time] INGEST_EVAL = _time=substr(_raw,0,10)   Configuration applied on the Splunk Search Head sh_azure_nsg_app/default props.conf #References: #https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview #https://community.splunk.com/t5/Splunk-Search/How-do-I-import-Azure-NSG-LOGs/td-p/396018 #https://community.splunk.com/t5/Getting-Data-In/How-to-extract-an-event-timestamp-where-seconds-and-milliseconds/m-p/428837 [mscs:nsg:flow] REPORT-tuples = extract_tuple_v1, extract_tuple_v2 REPORT-nsg = sub_res_nsg FIELDALIAS-mscs_nsg_flow = dest_ip AS dest src_ip AS src host AS dvc EVAL-action = case(traffic_result == "A", "allowed", traffic_result == "D", "blocked") EVAL-protocol = if(match(src_ip, "^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"), "ip", "unknown") EVAL-direction = case(traffic_flow == "I", "inbound", traffic_flow == "O", "outbound") EVAL-transport = case(transport == "T", "tcp", transport == "U", "udp") EVAL-bytes = (coalesce(bytes_in,0)) + (coalesce(bytes_out,0)) EVAL-packets = (coalesce(packets_in,0)) + (coalesce(packets_out,0)) EVAL-flow_state_desc = case(flow_state == "B", "begin", flow_state == "C", "continuing ", flow_state == "E", "end") transforms.conf [extract_tuple_v1] DELIMS = "," FIELDS = time,src_ip,dest_ip,src_port,dest_port,transport,traffic_flow,traffic_result [extract_tuple_v2] DELIMS = "," FIELDS = time,src_ip,dest_ip,src_port,dest_port,transport,traffic_flow,traffic_result,flow_state,packets_in,bytes_in,packets_out,bytes_out [sub_res_nsg] SOURCE_KEY = source REGEX = SUBSCRIPTIONS\/(\S+)\/RESOURCEGROUPS\/(\S+)\/PROVIDERS\/MICROSOFT.NETWORK\/NETWORKSECURITYGROUPS\/(\S+)\/y=\d+\/m=\d+\/d=\d+\/h=\d+\/m=\d+\/macAddress=(\S+)\/ FORMAT = subscriptionID::$1 resourceGroupName::$2 nsgName::$3 macAddress::$4 eventtypes.conf [mscs_nsg_flow] search = sourcetype=mscs:nsg:flow src_ip=* [mscs_nsg_flow_start] search = sourcetype=mscs:nsg:flow flow_state=B [mscs_nsg_flow_end] search = sourcetype=mscs:nsg:flow flow_state=E tags.conf [eventtype=mscs_nsg_flow] network = enabled communicate = enabled [eventtype=mscs_nsg_flow_start] network = enabled session = enabled start = enabled [eventtype=mscs_nsg_flow_end] network = enabled session = enabled end = enabled   Best Regards, Edoardo   ... View more

Hello, Do you mean the 200GB/day is for an 12vCPU/12GB RAM/900 IOPS Heavy Forwarder that is indexing locally and also forwarding to Indexers but not performing local searches? In this 200GB/day are you also including logs from internal indexes ( index=_* ) ? If so, what about an Heavy Forwarder with same specs that is not locally indexing? How many GB/day can process (internal and non internal logs)? Thanks a lot, Edoardo ... View more

Yes I know 🙂 Can we assume the same compression ratio? Or is there any official feedback on that somewhere in the docs? ... View more

Thank you so much, it works! ... View more

Same issue but I believe it is a problem related to user Role Capabilities / Indexes / Restrictions Tested among 2 different user Roles and one can run tstat query, the other cannot In my case they both have access to the indexes but one Role have some Search filter restriction   ... View more

Replying also here, referring to this post: https://community.splunk.com/t5/Getting-Data-In/Why-has-the-index-process-paused-data-flow-How-to-handle-too/m-p/633554/highlight/true#M108467   We ended up with this configuration: In server.conf [queue=indexQueue] maxSize=500MB In indexes.conf [default] throttleCheckPeriod=5 maxConcurrentOptimizes=2 maxRunningProcessGroups=32 processTrackerServiceInterval=0   In this way we have both the benefits: if we restart the cluster we don't have anymore the IndexWriter message during the normal running we don't have the HealthChangeReporter OR PeriodicHealthReporter messages anymore Thanks a lot for your suggestion! ... View more

Hi @hrawat_splunk  We ended up with this configuration: In server.conf [queue=indexQueue] maxSize=500MB In indexes.conf [default] throttleCheckPeriod=5 maxConcurrentOptimizes=2 maxRunningProcessGroups=32 processTrackerServiceInterval=0   In this way we have both the benefits: if we restart the cluster we don't have anymore the IndexWriter message during the normal running we don't have the HealthChangeReporter OR PeriodicHealthReporter messages anymore Thanks a lot for your suggestion! ... View more

Thanks for your reply! We have already followed the guide you provided and also the one listed here below. By the way I opened an Idea called "Virtualization and Performance guide for deploying Splunk" https://ideas.splunk.com/ideas/EID-I-1008   Here below some example of case-study we have followed:   https://core.vmware.com/resource/splunk-vmware-vsan   https://www.dell.com/community/s/vjauj58549/attachments/vjauj58549/storage-and-data-protection-wiki-ch/3903/1/h15604-splunk-enterprise-and-vmax-all-flash.pdf   https://www.delltechnologies.com/resources/en-us/asset/offering-overview-documents/products/storage-2/h15699-splunk-vxrail-sg.pdf   https://www.intel.com/content/dam/www/public/us/en/documents/reference-architectures/high-performance-data-analytics-with-splunk-brief.pdf   ... View more

Hello @hrawat_splunk  Really thanks for your reply. We applied the suggested configuration both on server.conf and indexes.conf So basically from my understanding the aim is to check for index throttling more often (throttleCheckPeriod=5), with only 1 splunk-optimize running over a bucket (maxConcurrentOptimizes=1), spawning several child processes over it (maxRunningProcessGroups=32) and checking every second if any other child process can be launched (processTrackerServiceInterval=0). Therefore the purpose is to concentrate all the optimize resources on a single bucket per time. What we observed is the following: if we put the Cluster Master in maintenance and stop an Indexer we do not see anymore the messages on the Monitoring Console. The Indexing Queue fill anyway to 100% on all remaining Indexers even if we have increase the size to 500MB, the Indexing Rate decrease from 15-20 MB/s to 1MB/s, but in short time (approximately few minutes) the problem is solved The side effect we observed is that since we applied the modification we see in Indexers _internal index the following messages (20-30 events per hour): 02-16-2023 17:15:33.556 +0100 INFO HealthChangeReporter - feature="Index Optimization" indicator="concurrent_optimize_processes_percent" previous_color=green color=yellow due_to_threshold_value=100 measured_value=1 reason="The number of splunk optimize processes is at 100% of the maximum. As a result, the index processor has paused data flow." 02-16-2023 17:15:47.753 +0100 INFO PeriodicHealthReporter - feature="Index Optimization" color=yellow indicator="concurrent_optimize_processes_percent" due_to_threshold_value=100 measured_value=1 reason="The number of splunk optimize processes is at 100% of the maximum. As a result, the index processor has paused data flow." node_type=indicator node_path=splunkd.index_processor.index_optimization.concurrent_optimize_processes_percent And we also see the following "original" message (1-2 events per day): 02-16-2023 14:45:38.658 +0100 INFO IndexWriter [12974 indexerPipe] - The index processor has paused data flow. Too many tsidx files in idx=_internal bucket="/xxxxxxx/xxxx/xxxxxxxxxx/splunk/db/_internaldb/db/hot_v1_1928" , waiting for the splunk-optimize indexing helper to catch up merging them. Ensure reasonable disk space is available, and that I/O write throughput is not compromised.   It seems to me the direction you given us is the correct one to solve the problem, in fact now it is fine if the Cluster takes just few minutes to recover, before was much longer. What we would like to improve is to avoid during the normal running the PeriodicHealthReporter and HealthChangeReporter messages that inform us the indexing has stopped. Do you think that we can increase the maxConcurrentOptimizes value to avoid that? I think in this way we could better balance the "brute force" over more buckets, probably we will loose something when an Indexer is stopped but we gain in normal running.   For reference here the indexes.conf specification: throttleCheckPeriod = <positive integer> * How frequently, in seconds, that splunkd checks for index throttling conditions. * NOTE: Do not change this setting unless a Splunk Support professional asks you to. * The highest legal value is 4294967295. * Default: 15 maxConcurrentOptimizes = <nonnegative integer> * The number of concurrent optimize processes that can run against a hot bucket. * This number should be increased if: * There are always many small tsidx files in the hot bucket. * After rolling, there are many tsidx files in warm or cold buckets. * You must restart splunkd after changing this setting. Reloading the configuration does not suffice. * The highest legal value is 4294967295. * Default: 6 maxRunningProcessGroups = <positive integer> * splunkd runs helper child processes like "splunk-optimize", "recover-metadata", etc. This setting limits how many child processes can run at any given time. * This maximum applies to all of splunkd, not per index. If you have N indexes, there will be at most 'maxRunningProcessGroups' child processes, not N * 'maxRunningProcessGroups' processes. * Must maintain maxRunningProcessGroupsLowPriority < maxRunningProcessGroups * This is an advanced setting; do NOT set unless instructed by Splunk Support. * Highest legal value is 4294967295. * Default: 8 processTrackerServiceInterval = <nonnegative integer> * How often, in seconds, the indexer checks the status of the child OS processes it has launched to see if it can launch new processes for queued requests. * If set to 0, the indexer checks child process status every second. * Highest legal value is 4294967295. * Default: 15   Thanks a lot, Edoardo ... View more

    @hrawat_splunk thanks for the update we have the same exact issue. I see you mentioned it has been fixed with 9.1, do you mean 9.0.1? The latest available is 9.0.3 We are on prem with 9.0.2 and still facing it despite we already put the indicated set-up in indexes.conf see my question here https://community.splunk.com/t5/Splunk-Enterprise/The-index-processor-has-paused-data-flow-How-to-optimize/m-p/629520 Do you suggest increasing maxRunningProcessGroups?   ... View more

Exactly, I confirm in our environment the issue was mainly due to thruput constraint, but we put 25 MB/s instead of unlimited. The only strange thing that didn't allowed us to quickly identify the root cause was that, for the windows events locally generated by the WEC server itself, the Splunk Universal Forwarder had no delay collecting them. The only delay was observed on forwarding with the Splunk Universal Forwarder the events stored by the Windows Event Collector (WEC) coming from the other machines through Windows Event Forwarding (WEF).   limits.conf [thruput] maxKBps = 25600     To understand the thruput limit in your environment you can use this query (stay quite higher than the maximum you observe)   index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) hostname=your_WEC_host | timechart minspan=30s max(eval(tcp_KBps)) as "KB/s", max(tcp_eps) as "Events/s"     ... View more