Splunk scheduled weekly report used to take 30 mins, now takes more than 20 hrs

[swamysanjanaput]

Hello Community,

We have a scheduled weekly report that used to take 30 mins, now takes more than 20 hrs to complete and when we inspected search job we could see few search components taking more duration. Is there any way out to identify what might be the root cause or what might have caused this search to run for loner duration? Any suggestions are appreciated!

The search has completed and has returned 3,664 results by scanning 438,673,166,793 events in 73,635.129 seconds~20.4542025 hrs
Duration (seconds) Component Invocations Input count Output count
205,560.16 command.search 4,020,759 7,580,746,540 9,809,797,326
256,005.83 command.tstats 2,990,397 39,155,807 39,159,471
255,889.30 dispatch.stream.remote 1,495,082 ----- 12,281,759,678

Following are the completion times

Nov 4th - 35 mins
Nov 11th - 35 mins
Nov 18th - 30 mins
Nov 25th - 35 mins
Dec 2nd - 29 mins
Dec 9th - 32 mins
Dec 16th - 34 mins
Dec 23rd - 36 mins
Dec 30th - 35 mins
Jan 6th - 35 mins
Jan 13th - 7h 5m
Jan 30th - 12h 58m
Jan 27th - 18h 35m
Feb 3rd - 35hr 45m

[gcusello]

Hi @swamysanjanaputta,
usually low search performaces are related to number of available CPU and storage performaces (Splunk requests at least 800 IOPS!).

Another usual problem is when you have in a search one or more join or transaction commands that are very slow for their nature: in this case you shuld try to replace these commands with a different search (e.g. using stats).

But if you have very many data, there could be a problem that you can solve with a different approach:

• you could schedule your search to extract results on a limitated period (e.g. 24 hours) by night, when your system has less load;
• save these results in a summary index;
• then run your search on the summary index that's quicker!

Ciao.
Giuseppe