Splunk Search

Lookups to files larger than max_memtable_bytes report file only contains a header row

joebensimo
Path Finder

With Splunk v5 and v6, I have not been able to get lookups to work with CSV files that are larger than max_memtable_bytes.

When attempting to lookup, input, or output to a lookup file that is larger than max_memtable_bytes, I get an error stating that the file is empty.

For example:
Empty csv lookup file (contains only a header) for table 'agenthash.csv': /opt/splunk/etc/apps/search/lookups/agenthash.csv

In the past, I've worked around this (as advised by support) by increasing max_memtable_bytes. However, I now have some lookups that are larger than most, and some that are at risk of growing to be larger than max_memtable_bytes.

The documentation says that Splunk will index larger files on disk, but I've not yet been able to get this to work. How can I use huge lookup files?

Tags (3)

haley_swarnapat
Path Finder

If you are using Windows, there is a workaround (not real solution, but it should solve your problem)

From your start menu type and search for "ODBC Data Sources"
Create a System DSN
Add "Excel Files" data source
Choose your CSV file
Now the CSV file becomes accessible via ODBC Driver, voila!

Use Splunk DBLookup to fetch data from the DSN

0 Karma

joebensimo
Path Finder

This continues to be a problem. It appears that Splunk's functionality to index large lookup files on disk has been broken for over a year. Is this broken? Or is there something special that needs to be done to make it work?

dshpritz
SplunkTrust
SplunkTrust

What version of Splunk are you running?

0 Karma

cramasta
Builder

Whats larger than most? What do you have max_memtable_bytes set to?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...