what is the most efficient way to achieve this.
I run search #1 that populates the lookup table file with data.
Then search #2 will search for values a specific field in the lookup table and only reports events that are NOT a match for anything already in the lookup table.
Finally I append the results of the second search to the same lookup table. So in the end my lookup file will now have 1 list of unique entries combined from 2 different searches.
Is that possible? Otherwise , what would be the most efficient way?
Well, starting from this:
http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/
You could probably achieve something similar to your wishes. I have created a search (no access to it at the moment - could post later) which - in pseudo search language - works like this for maintaining a list of userid's;
sourcetype=xxx userid=* NOT [search |inputlookup userid_file | fields + userid] | fields + userid | outputlookup append=t userid_file
OR this (don't remember)
sourcetype=xxx userid=* | fields + userid | inputlookup append=t userid_file | dedup userid | outputlookup userid_file
EDIT: several small fixes.
Good luck
/Kristian
please explain it with some data..