I have a requirement to provide histograms of performance through Splunk. Essentially we have a field (for example Page_Load_Time), and we need to find out how may entries for that field (on a particular search) fall into certain fixed categories - e.g. <200ms,200ms-2s etc
To achieve this I've written a custom search command - splitbins
import splunk.Intersplunk
import sys
def sortValueToBin(fieldValue,listOfBins):
binNumber = 1
for binRoof in listOfBins:
if fieldValue < float(binRoof):
return "Bin-" + str(binNumber)
else:
binNumber +=1
return "Bin-" + str(binNumber)
fieldToSplit = sys.argv[1]
listOfBins = sys.argv[2:]
eventsDict,dummyResults,dummySettings = splunk.Intersplunk.getOrganizedResults()
for event in eventsDict:
# Check its a number we're trying to split on, otherwise skip the event
try:
fieldValue = float(event[fieldToSplit])
except:
continue
event["Bin_Number"] = sortValueToBin(fieldValue,listOfBins)
splunk.Intersplunk.outputResults(eventsDict)
This is then being run through a search command like this:
index="some_indexname" host="some_hostname" some_field="some_otherterm" | splitbins Page_Load_Time 200 2000 4000 8000 | chart count(Bin_Number) over some_other_field by Bin_Number | fields some_other_field Bin-1 Bin-2 bin-3 Bin-4 Bin-5
...and it works fine if the events passed by the initial search terms is in the thousands. However, as the number of events grow - two problems occur:
Results stop being produced once the total number of events processed goes over 50,000
The search is S-L-O-W. For example 20 minutes for 250K events. If I write the splitbins code to take a direct dictionary with some random results, it can process hundreds of thousands of events in less than a second: so there is nothing innately slow about the splitbins code.
I've tried to adjust everything in limit.conf that is set to 50000 to be a higher number with no change to the events processed. I've tried adding in a fields pipe after the initial search string to try and slim the search objects down earlier, and it is still slow.
Running v4.1.2 on Windows, with plenty of spare CPU and memory.
Any ideas?
Thanks
... View more