Dashboards & Visualizations

Performance issue when using transaction? Dashboard not always loading properly.

the_wolverine
Champion

I have noticed that when I use a transaction-based search in my dashboard panels, the performance is not so great and my panels do not always load (perhaps the search times out)? Other than summarizing these searches, is there anything I can do to improve the performance of my transactional searches?

For example:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring" | transaction host maxspan=10m | where linecount>9 | stats count host (over past 24 hours)

Perhaps there is a way to lengthen the timeout (if that's the problem that I'm encountering.)

0 Karma

MartinHarper
Path Finder

Another thing you can do, apart from summarizing the searches, is putting them into a saved search and scheduling them to run every hour. That way the dashboard will display the saved search results near-instantly, rather than kicking off a new instance of the search that could take several minutes to return.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Seems to me you could get similar (not identical) results much faster (and also taking advantage of map-reduce much more effectively, assuming you have multiple indexers) with:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring"  
| bucket _time span=10m
| stats count by _time,host
| where count > 9
| stats distinct_count(host)

You'd have to adjust the counts and span from what you have since it will use a fixed rather than sliding time window (e.g., change the threshold to 6, or the span to 20 minutes and the threshold to 15), but the metric is unlikely to be that hard-defined and the performance tradeoff may be worth it.

0 Karma

the_wolverine
Champion

Thanks for your response, Gerard. Your search appears to run faster.

My search, otherwise, works fine for me giving me the data in the format I want and on a sliding window basis. I use this to alert on potential brute force attack on a system and can't limit it to a fixed time window. Is there some way to bypass any timeout I might be hitting in the UI?

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...