All Apps and Add-ons

NAS or SAN issue?

a212830
Champion

Hi,

I'm battling some performance issues since upgrading from 6.4.1 to 6.5.5 and I noticed that we are getting the dreaded "underlying storage issues" messages from Splunk. I'm using SHP (yeah, I know...), which uses NAS. The messages seems to point to the SAN (where the binaries are loaded, and not the NAS). Is that acccurate? I need to know which support group to research the "storage issues".

Configuration initialization for /apps/splunk/etc took longer than expected (1121ms) when dispatching a search (search ID: scheduler_adminSplunkGov_RMD576fcb4cded0d0548_at_1501394400_6023); this typically reflects underlying storage performance issues

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Another answer from a peer of mine...

I am going to assume that the SAN is fast (which MUST be verified despite the storage admins claim):
- SHP is deprecated as you know. I strongly recommend making it a priority for the customer to use SHC
- NAS is probably bad option to use due to I/O issues. I have seen a customer make it work but at greater expense (dedicated hardware/switches and SSD/FLASH)
- It’s possible that a binary (on a SAN LUN) may generate the error but the origin of the problem is in the actual data storage (SAN). Enabling splunk debug or using OS tools like dtrace may shed some light on the issue

http://www.slashroot.in/san-vs-nas-difference-between-storage-area-network-and-network-attached-stor...

…Although NAS is a cheaper option for your storage needs. It really does not suit for an enterprise level high performance application. Never ever think of using a database storage (which needs to be high performing) with a NAS. The main downside of using NAS is its performance issue, and dependency on network(most of the times, the LAN which is used for normal traffic is also used for sharing storage with NAS, which makes it more congested)…

0 Karma

mdonnelly_splun
Splunk Employee
Splunk Employee

In SHP the etc/apps directory is typically shared via NAS.

Long term, your solution is to move from SHP + NAS to Search Head Clustering on local disk. But your "yeah, I now..." tells me you already know that.

In terms of troubleshooting this issue, there may in a couple different areas:
- the NAS / LUN performance
- the network performance between your SHP members and the NAS device
- If your search heads are running as VMs, the VMware environment may ALSO be a factor
- and Double check your NTP time sync!

This is a great time to mention that there are a number of ways that Splunk can be used to monitor the performance of the NAS and VMware environments, so that you and others don't need to escalate a call to the storage team to see if there's a storage issue, don't need to escalate to the VMware team to see if VMs are performing well and configured correctly.

Splunk IT Service Intelligence is a great way to tie all of these together simply and easily, so whether you're trying to troubleshoot Splunk or a mission critical application stack, everyone sees the information all in one place. ITSI provides a fantastic set of functionality.

0 Karma

jpagan_splunk
Splunk Employee
Splunk Employee

Hello a212830!

I'm sure you're already aware, but it's still worth mentioning for the benefit of others, SHP (Search Head Pooling) is deprecated as of 6.2.

Now on to the error message... On the surface, the error message does appear to point to SAN performance as the primary performance concern. This assumes that $SPLUNK_HOME points to /apps/splunk/ and this is your SAN filesystem. However, other resource constraints could be causing systemic issues and this error could just be a red herring. To confirm, I would run Bonnie++ to test storage performance on both the NAS and SAN filesystems.

Check out the awesome Bonnie++ app on Splunkbase -> https://splunkbase.splunk.com/app/3002/

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...