How to update a lookup table using a scheduled search by appending results, not replace the old data?

[harshal_chakran]

Hi,
I have one scheduled search which saves the output in a file "filename.csv" at specific interval of time.

index="myindex"|........|.....|outputlookup filename.csv

But what is happening now is that the complete file content is replaced with the new one, hence losing the old data from the file. I want to append the search result to the data already present in filename.csv

Please Help...!!!

[Runals]

As vganjare mentions it would be handy to get an idea of what you are doing with your lookup. At any rate I think what you are looking for is when you do a | lookup somefile.csv you need to put append=true. As an example here is the first of 2 queries used to track systems that stop sending logs. This one tracks the last time a host sent in logs (runs every 4 hrs) and others run every 8 and run a check against the last_seen field.

index=foo | eval host=lower(host) | rex field=host "(?<host>(^[^0-9]\S[^\.]+)|(^[0-9]\S+))" | stats max(_time) AS last_seen by host | inputlookup append=T hosts_list.csv | stats max(last_seen) AS last_seen by host | eval right_now = now() | eval time_diff = right_now - last_seen | where time_diff < (86400 * 3) |  table host last_seen | outputlookup hosts_list.csv

Relative to your question I'm getting results from a query, adding those results to the csv, manipulating the results, and then writing the results back to the csv. The 2 almost back to back stats commands are because once you've appended the results to the existing csv most systems will have 2 lines and I'm only interested in keeping the latest.

BTW I do this as a csv because if a system is decommissioned I can simply remove it from the list.

[vganjare]

You can try something like:

|inputlookup filename.csv | append [your new search] | filter the duplicate events if required | outputlookup filename.csv

Thanks!!