Getting Data In

Datamodel Acceleration: How to reduce high memory usage?

olivier_ma
Explorer

Hello,

I'm facing a high memory usage on all of the 3 indexers when I try to accelerate a datamodel, even for 1 day acceleration.
After investigation, high memory usage is due to searches run for datamodel acceleration.
Our indexers have 20Gb of RAM. The accelerated datamodel is about proxy logs (~60Gb / day) and it's an root-event based datamodel.

How can I debug this issue ?
Did I miss something on datamodel acceleration ?

Thanks

0 Karma

woodcock
Esteemed Legend

Configure BOTH index= AND sourcetype= in your cim_<DMA>_index macros.
Configure tags whitelist.
Check out the known issues page of the release notes for your release and implement the workarounds listed for the DMA bugs for your version (there are some DOOZIES).
Get more RAM for your Search Heads and Indexers and RESERVE IT EXCLUSIVELY if in a VM.
Deploy more Indexers.

0 Karma

skalliger
SplunkTrust
SplunkTrust

What are your specs of the ES SH and the indexers? Are your DMAs completing in time?
Did you restrict the DM to the specific index(es)? You can also try increasing the number of instances assigned to the DMA, this can also be done via UI (or acceleration.max_concurrent in datamodels.conf).
Did you set skewing (called allow_skew ) for scheduled searches and DMA already?

If you still run into issues after doing some tuning, you might wanna contact support/PS about further improvements.

Skalli

0 Karma

olivier_ma
Explorer

I don't have ES. Here is specs
1 SH (RHEL):
8 CPU, 12 Gb of RAM, Splunk 7.0.2
3 Indexers (RHEL)
16 CPU, 24 Gb of RAM, Splunk 7.0.2

This DMA didn't complete as indexers systems are killing splunk processes for high memory usage.
But other DMAs (on other DM) work perfectly.

All accelerated DMs are restricted to specific index.
I'm pretty afraid about increasing the number of concurrent acceleration as enabling this one (proxy DMA) is already causing high memory usage. Currently it is set to 1.

I didn't know about skew settings. I'll take a look.
Thanks for the hand. I'll let you know if it helps 🙂

0 Karma

skalliger
SplunkTrust
SplunkTrust

How much are you ingesting per day and what is the size of the mentioned DM?
If only the big DMA doesn't get completed in time right now, skewing could help but in the long term you might wanna add another indexer to distribute the load further.

Skalli

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...