I have the following search query which does what I'd like:
sourcetype=my_log
| eval adj_request_id = if(isnotnull(original_request_id), original_request_id, request_id)
| eventstats count as request_id_count by adj_request_id
| eval validated=if(request_id_count > 1, "true", "false")
The query works, but unfortunately my log is huge and slow on its own. Adding the event stats command makes it basically unusable. Is there a way to run this say every day at midnight on the previous days data and have it permanently add the field so that the eventstats
call isn't needed every time I want to access the validated field?
Or is there a way to just dramatically speed up this query, or a different and faster way of accomplishing this? Any help would be very much appreciated!
This sounds like a job for a summary index. Run your search overnight and write the results to a summary index. Then just fetch the validated field from the summary index when you need it. See http://docs.splunk.com/Documentation/Splunk/6.3.1/Knowledge/Usesummaryindexing for more information.
Conceptually, the summary index seems similar to what I want, but it looks like eventstats cannot be summary indexed...