So I have a search that can be run that returns 12 events over a 60 minute period from a single known source. I would like to run a general query that correlates all other events from all sources that also occur 12 times in the 60 minute period.
The hypothesis being that the error I am searching for may also have events somewhere that correlate, however it's difficult to go through dozens of source-types and events manually doing this.
Just widen your base search.
If originally you were doing something like:
index="myindex" sourcetype="mysourcetype"
Try:
index="myindex" | transaction maxspan=1m maxpause=5s
This will take all events from your index and make transactions of 1 minute length with a maximum time between events of 5 seconds.
If your search needs to cross multiple indexes, just list all the indexes with OR statements out before the pipe:
index="myindex1" OR index="myindex2" OR...
You could have a look at the sparkline feature for stats
or chart
.
earliest = -1h | stats sparkline c by sourcetype
which could give you a quick overview of things that seem to be happening at the same time.
http://docs.splunk.com/Documentation/Splunk/5.0.5/Search/Addsparklinestosearchresults
/K
All of the examples are assuming transform is being used on a single source, which I can do easily, however if I want to try to correlate on other events without common fields just by time, im drawing a blank
Look at the maxspan, maxwait, and similar attribute of the transaction command. One of those is probably what you are looking for
Yes, im trying to look into transaction, but there is no common field to be aggregated other than a +- 1 second interval, but im not sure how to employ that.
For example I have an error log that 5% of the time returns a stack trace. I generate a search to isolate those events. I want to be able to run something that figures out what other events across all sources in the same index correlate most closely to those "5%" events based on the timestamp.
How about using stats by sourcetype or eventtype?
You can run two separate searches:
<base search...> | stats count by sourcetype | where count = 12
OR
<base search...> | stats count by eventtype | where count = 12
This won't give you the correlation you are looking for, but it will give you only sourcetypes or eventtypes that had 12 events in your time frame.
For correlation you may want to look into transactions, after you have identified a field that you can use to correlate all the events together.