Monitoring Splunk

Splunk Server performance poor

meenal901
Communicator

Hi,

I have a windows-based splunk instance. The server capacity is 4GB RAM. I am indexing around 50MB data per day.
After 1 month, the data loaded into splunk is around 4GB and now when i am loading my dashboad with 2 charts on it, it takes a lot of time.
Also sometimes the dashboard does not show the charts as well, although plain search returns the expected results.

Is there some way to check if the server is, may be, getting overloaded. What to do in such situation?

Thanks,
Meenal Luktuke

Tags (1)

lguinn2
Legend

First, are you following the sizing recommendations and other capacity planning info in the Installation manual? My initial thought is that your server may be undersized - the server size recommendations are

  • Intel x86 64-bit chip architecture
  • 2 CPUs, 4 cores per CPU (8 cores total), at least 2.5 Ghz per core
  • 8 GB RAM
  • Standard 1 Gb Ethernet NIC, optional 2nd NIC for a management network
  • Standard 64-bit Linux or Windows distribution

How many CPUs do you have? How full is your disk - and how fast is it? Splunk wants disks that can deliver 800 IOs per second. Is this a virtual machine or a physical machine?

Second, have you checked out the Splunk community wiki? It has a variety of troubleshooting information, including this general overview of performance troubleshooting.

I would look at the splunkd log (you can search it via index=_internal) to see if there are any errors or warnings being reported. The documentation, the wiki and this forum can help you understand any errors/warnings from splunkd. I would also look at your basic server performance indicators - what do CPU, memory, and disk IO statistics look like?

Finally, are you running scheduled searches or alerts? Real-time searches? Some apps run searches in the background, so be sure to check all the apps. What is the time range for the searches on the dashboards?

Usually, if your server is overloaded, your best option is to add another server. But not always. If your server is below the Splunk specification, you might first try to upgrade your server.

lguinn2
Legend

Thanks for the comment re: timechart - good point!

0 Karma

Drainy
Champion

I'll just add this as a comment as lguinn has thrown up a pretty comprehensive answer 😉 To me it sounds as if your CPU is under-spec. I am going to make an assumption that when you say charting you mean timechart, in this case Splunk needs to bucket the data into different time spans and this can be CPU heavy on an under-spec'ed CPU. In the same way this would also explain the slowdown overall you're experiencing

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...