I have a lookup file with 50,000 records. When I want to do a search, it takes a lot of time to find my results. Is there a way to get faster and better searches result?
Hi @badoomi,
As a csv lookup file starts getting more and more entries it's recommended to move the csv entries to a kvstore
.
Have a look here, it's a great document explaining why use a kvstore :
http://dev.splunk.com/view/webframework-developapps/SP-CAAAEY7
Some of the advantage described there for kvstore
vs csv
are :
- Enables per-record insert/updates
("upserts").
- Allows optional data type enforcement
on write operations.
- Allows you to define field
accelerations to improve search
performance.
- Provides REST API access to the data
collection.
Also it's fairly easy to configure and use, in case you haven't done so before you can follow this guide :
https://docs.splunk.com/Documentation/Splunk/7.3.0/Knowledge/ConfigureKVstorelookups
Cheers,
David
@badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup
By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not.
In essence you would need to test out stats first then lookup
vs lookup first and stats next
.
Do share your current SPL for community members to assist you better with your use case.
@niketn wrote:@badoomi, optimizing lookup search may not be straight-forward without knowing your SPL and Splunk Infra ( as to how many Indexers you have got). However you can refer to following Splunk Documentation for one of tip to optimize lookup
By default lookup command runs with argument local=true which means it is executed on Search Peer not on Search Head. If you have multiple indexers and your SPL till the lookup command have only streaming commands then there would be an advantage of this otherwise not.
In essence you would need to test out stats first then lookup vs lookup first and stats next.
Do share your current SPL for community members to assist you better with your use case.
I think there may have been a typo this this original answer as the lookup command has local=false set by default - source: https://docs.splunk.com/Documentation/Splunk/9.1.1/SearchReference/Lookup
localSyntax: local=<bool>
Description: If local=true, forces the lookup to run on the search head and not on any remote peers.
Default: false
i have one indexer and one forwarder.i create a automatic lookup.my search is
index=fw or index=waf | where ip=m_ip | stats count by src,category
Can you give an example of the search you are attempting on the lookup.
ie | lookup or | inputlookup