Hitting eventstats memory limit. Is there another way to form my search?

[jedatt01]

I'm seeing the following error below when trying to use eventstats on a large lookup table (about 500,000 rows).

06-24-2016 15:44:04.735 ERROR StatsProcessor - Reached limit max_mem_usage_mb (200 MB), results may be incomplete! Please increase the max_mem_usage_mb in limits.conf .

Getting the max_mem_usage_mb increase will take a long time as I'm a user in a large enterprise environment that would require testing first.

In the lookup table, I have records of machine check-ins to an DM system. I'm trying to create a search that will look for the latest check-in in the lookup and remove the older check-ins. My current search below is not completing because of the memory limit.

| inputlookup mdm_master | eventstats max(MDM_Last_Check_in_epoc) as last_check by MDM_MAC_Address MDM_Server | where MDM_Last_Check_in_epoc = last_check | table MDM* | outputlookup mdm_master

Can anybody think of a more efficient way to do this that will take less memory?

[maciep]

Not sure how these things work under the hood, so this may be as or more inefficient, but you could try using sort/dedup.

| inputlookup mdm_master 
| eval mdm_mac_server = MDM_MAC_Address." ".MDM_Server
| sort mdm_mac_server - MDM_Last_Check_in_epoc
| dedup mdm_mac_server
| table MDM*
| outputlookup mdm_master

Also, again not sure how efficient, but you could try to use a stats instead of eventstats, and then lookup or join the results to the lookup again, keep the ones that match output the lookup again?

[jedatt01]

must be hardcoded somewhere. I tried sort 0 and also sort 500000 and it still tells me it's being limited to 10k results.

[maciep]

interesting...maybe it's in limits somewhere.

What about looking up from the lookup? Something like this?

| inputlookup mdm_master 
| tstats max(MDM_Last_Check_in_epoc) as MDM_Last_Check_in_epoc
| lookup mdm_master MDM_Last_Check_in_epoc
| table MDM* 
| outputlookup mdm_master

[jedatt01]

Unfortunately, sort has a 10,000 event limit. No good there either.

[maciep]

Oh, whoops....you can use sort 0 to remove the default limit, like

| sort 0 mdm_mac_server - MDM_Last_Check_in_epoc

[aljohnson_splun]

I sort of wonder if you could use streamstats and just get the last calculation - but to be honest I don't really know. Could be worth a try ?

[woodcock]

2 options, both are higher-level things:

1: Setup a Data Model and accelerate it then convert your search to tstats.
2: Setup a Summary Index.