Splunk Search

How many inputlookup files can be read in a splunk query

hanikawadhwa
Explorer

Hi Splunkers,

I have been given a requirement where I need to read more than 10k input lookup files to get some results. My question: is it advisable or feasible to read this number of input lookup files?
scenario: every lookup files showing the operational hours of each trains for last 10 days.

(mon)9-10 (tues) 11-12 (wed )8-9 (thu)7-6 (fri) 1-2 (mon) 3-3:30 (tue )10-18 (wed)7-10 (thur) 9-18 (fri) 4-9

query requires to read all these input lookup files to consider operational hours of all the 10k trains.

Is there any other way to read all these files in a single query

TIA

0 Karma

niketn
Legend

@hanikawadhwa seems a better use case for Single aggregated KV Store if at any given day you need to have data for 10K trains for 7 days i.e. 70K results.

Which implies inputlookup fileXXXXX.csv | outputlookup consolidatekvstore kind of scheduled search to read from lookup file and populate consolidated KVStore. There has to be another search to remove entry older than 7 days.

If this is a CSV file, other option would be to simple read the CSV file from the folder location where they are being written and upload to an index so that all this can be taken care automatically. With indexed_extraction, the tstats based query on index would perform way better on 70K results and even more as you can retain older data as per your disk utilization or retention requirement (even 1 year or more).

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

hanikawadhwa
Explorer

Thank you so much. I understand that uploading to an index is a better way.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...