With Splunk v5 and v6, I have not been able to get lookups to work with CSV files that are larger than max_memtable_bytes.
When attempting to lookup, input, or output to a lookup file that is larger than max_memtable_bytes, I get an error stating that the file is empty.
For example:
Empty csv lookup file (contains only a header) for table 'agenthash.csv': /opt/splunk/etc/apps/search/lookups/agenthash.csv
In the past, I've worked around this (as advised by support) by increasing max_memtable_bytes. However, I now have some lookups that are larger than most, and some that are at risk of growing to be larger than max_memtable_bytes.
The documentation says that Splunk will index larger files on disk, but I've not yet been able to get this to work. How can I use huge lookup files?
If you are using Windows, there is a workaround (not real solution, but it should solve your problem)
From your start menu type and search for "ODBC Data Sources"
Create a System DSN
Add "Excel Files" data source
Choose your CSV file
Now the CSV file becomes accessible via ODBC Driver, voila!
Use Splunk DBLookup to fetch data from the DSN
This continues to be a problem. It appears that Splunk's functionality to index large lookup files on disk has been broken for over a year. Is this broken? Or is there something special that needs to be done to make it work?
What version of Splunk are you running?
Whats larger than most? What do you have max_memtable_bytes set to?