Is it possible to keep a running collection of events whenever a search is run?

[mpuckettsc]

I'm a bit confused on how to do something in Splunk that I would think is fairly obvious.

I have several million firewall logs (palo alto) where I want to be able to pull a report for the last 3 days of data which has been taking about 10 minutes to complete. I'd like to be able to keep a running collection of events that match so that whenever I pull the report I'm only searching for events that happened since last time it was pulled - not re-searching the entire 3 day window.

This seems like exactly like what collect and summary indexes are used for, but everything I'm reading about summary indexes wants to use statistics commands that I don't need, and collect strips out all of my fields making the data a bit useless.

What am I missing?

This is the search if it matters:

index=paloalto AND sourcetype=pan:traffic AND action=allowed AND src_class=private AND dest_class=unknown AND NOT dest_location=US AND bytes>5000000| eval megabytes=((bytes/1024)/1024) | table _time,src_ip,src_location,dest_ip,dest_location,megabytes | sort megabytes desc

[jpolcari · Accepted solution]

Are you looking to use the stored results to run further searches after? This might be a good candidate for a lookup table? You can append the results to a table each time that the search is run. You would use the outputlookup command to store the table of results in the lookup each time. You could schedule this as well.

[mpuckettsc]

I think I follow where you're going with that. The only search I would need to do on the events after would be to curtail the results to the last 72 hours. Is that going to be possible with a lookup table? I've only used look up tables once for something completely different.

[jpolcari]

Ahh ok. I think I understand a bit better what you are trying to do then. Depending on the number of results, this could still work for you. You could schedule this to run at a smaller interval(say every couple hours or so) and keep appending. You are able to use | lookup *lookupname* to view what is contained within the lookup table. It would eventually grow to contain a lot of results possibly though.

If you are just looking to get the results faster or not have to manually run the search every time. You could save this as a scheduled report and have it automatically run at a particular time. That way, when you view the report you will be shown the results of the last time it ran.

[mpuckettsc]

Thank you, it looks like I'm going to be able to make this work just fine with lookup tables.

The problem with the scheduled report route was we have a few of these that end up in a dashboard that take a while. Each time we would pull up the dashboard one of the reports would still be running so there would be a delay.

I think what I can do to keep it from getting too long is run a long interval search overnight when no one is using the system that overwrites and then append to it through out the day.

Also for anyone else that finds this, was able to search the lookup table and filter by my timestamp (mytime) with:

|inputlookup foreign_transfers.csv | eval _time=strptime(mytime,"%Y-%m-%dT%H:%M:%S.%6N") | where _time >= relative_time(now(), "-2h")

The string in the strptime function needs to match the format of your timestamp.