How can I improve my search performance? (I think time conversion is why my search is slow)

[szimmer661]

I'm running the following query:

index=ironstream MFSOURCETYPE=SMF110 SAPPLID=CSFBTP* 
| bin _time span=1d 
| eval cpu=USRCPUT_MICROSEC/1000000
| eval RegionType=if(SAPPLID="CSFBTPC1","Legacy","Laurus")
| eval IntervalDate=strftime(_time,"%Y/%m/%d")
| chart count(TRAN) as "Total Transactions" sum(cpu) as "Total CPU" over IntervalDate by RegionType

It passes millions of records and I suspect converting microseconds to seconds for each record may be the major reason. Is there a way to allow the search to sum up CPU time in microseconds and perform the division one time prior to display in the chart?

Is there a performance blog or other available resource that would help me in writing efficient searches?

[richgalloway · Accepted solution]

Look at the job inspector for the search (select "Inspect job" from the Job dropdown). It will tell you where the most time is spent processing your query.
I doubt converting microseconds to seconds is taking that much time. Depending on the size of your index and how many indexers are sharing the work, the most time may be spent reading events.
I would, however, move the bin command to just before chart.

[szimmer661]

Thanks for the prompt reply.

I just thought that running the eval to convert ms to seconds 13 million times as opposed to once prior to the chart command would have a cumulative effect on the overall performance. Moving the bin command to just before chart did seem to help some.

From the job inspector, it looks like most of the time is spent in dispatch.stream.remote. Since I have no idea what that is, I'll capture the job inspector output and the search log and send them to our Splunk admin.