Hi,
I have one scheduled search which saves the output in a file "filename.csv" at specific interval of time.
index="myindex"|........|.....|outputlookup filename.csv
But what is happening now is that the complete file content is replaced with the new one, hence losing the old data from the file. I want to append the search result to the data already present in filename.csv
Please Help...!!!
As vganjare mentions it would be handy to get an idea of what you are doing with your lookup. At any rate I think what you are looking for is when you do a | lookup somefile.csv you need to put append=true. As an example here is the first of 2 queries used to track systems that stop sending logs. This one tracks the last time a host sent in logs (runs every 4 hrs) and others run every 8 and run a check against the last_seen field.
index=foo | eval host=lower(host) | rex field=host "(?<host>(^[^0-9]\S[^\.]+)|(^[0-9]\S+))" | stats max(_time) AS last_seen by host | inputlookup append=T hosts_list.csv | stats max(last_seen) AS last_seen by host | eval right_now = now() | eval time_diff = right_now - last_seen | where time_diff < (86400 * 3) | table host last_seen | outputlookup hosts_list.csv
Relative to your question I'm getting results from a query, adding those results to the csv, manipulating the results, and then writing the results back to the csv. The 2 almost back to back stats commands are because once you've appended the results to the existing csv most systems will have 2 lines and I'm only interested in keeping the latest.
BTW I do this as a csv because if a system is decommissioned I can simply remove it from the list.
You can try something like:
|inputlookup filename.csv | append [your new search] | filter the duplicate events if required | outputlookup filename.csv
Thanks!!