We have 9,255,277,001 events indexed for 90 days of hot/warm data. We need to run on a single Splunk instance. Our server has 24 CPUs and 256GB memory and disks are to Splunk IOPS specifications.
When we run this search on the system for 30 days it takes 6+ minutes.
| tstats count AS totalSessions
FROM datamodel=nmdm_flow
WHERE nmds_app_flow.c_user="" AND nmds_app_flow.d_name="" AND nmds_app_flow.app_name="" AND nmds_app_flow.dest_ip4_Country="" AND (nmds_app_flow.event!="Update")
| eval totalSessions = tostring(totalSessions, "commas")
CPU and memory usage on the server is minimal during this search.
We are hoping that by changing some configuration settings in limits.conf (or other conf files) we could make better use of the CPU/Memory hardware during our searches. For example, we have tried changing max_mem_usage_mb from default of 200 and are seeing improvements in the above search.
hey
have a look at this option:
[thruput]
maxKBps = <integer>
* The maximum speed, in kilobytes per second, that incoming data is
processed through the thruput processor in the ingestion pipeline.
* To control the CPU load while indexing, use this setting to throttle
the number of events this indexer processes to the rate (in
kilobytes per second) that you specify.
* NOTE:
* There is no guarantee that the thruput processor
will always process less than the number of kilobytes per
second that you specify with this setting. The status of
earlier processing queues in the pipeline can cause
temporary bursts of network activity that exceed what
is configured in the setting.
* The setting does not limit the amount of data that is
written to the network from the tcpoutput processor, such
as what happens when a universal forwarder sends data to
an indexer.
* The thruput processor applies the 'maxKBps' setting for each
ingestion pipeline. If you configure multiple ingestion
pipelines, the processor multiplies the 'maxKBps' value
by the number of ingestion pipelines that you have
configured.
* For more information about multiple ingestion pipelines, see
the 'parallelIngestionPipelines' setting in the
server.conf.spec file.
* Default: 0 (unlimited)
let me know if this helps!