Installation

Can't use CIFS for coldPath "unusable filesystem" - Splunk 6.1.2

cam343
Path Finder

Hello,
We have a splunk cluster running 6.1.2 - I recently tried to configure CIFS storage (via mountpoint) for cold and frozen buckets.
After setting the required config in indexes.conf, I restarted Splunk only to receive the following error:

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
coldPath='/mnt/splunkArchive/dr/colddb' of index=dr on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

Running 'locktest' manually I get:

[root@splunk901 bin]#  export SPLUNK_DB="/mnt/splunkArchive/dr/colddb/"; ./locktest
Could not create a lock in the SPLUNK_DB directory.
Filesystem type is not supported: buf.f_type = 0xff534d42
If supporting this filesystem type is important to you, please file an Enhancement Request with Splunk Support with the fs info number listed.

According to http://docs.splunk.com/Documentation/Splunk/6.1.2/Installation/Systemrequirements#Supported_file_sys...
CIFS is supported. Therefore is it possible my implementation of CIFS is not standard and doesn't for example support file locks (or similar)

Any help is appreciated....

I also have a second question if the NAS server and can be configure to use NFS instead of CIFS, will the same problem occur, due to the underlying filesystem? or does NFS abstract from the underlying filesystem?

Item specifics:
Redhat 6.5 64 bit
Isilon NAS running OneFS

Many Thanks
Cam

Tags (4)

jrodman
Splunk Employee
Splunk Employee

We just have a list of known filesystems that locktest has a strategy for, and the code for CIFS isn't set up, which tells you no one has done it before, and it probably was never tested.

However, hopefully the exposed semantics for CIFS on unix are similar enough to those on windows that the default lock strategy works acceptably.

cam343
Path Finder

According to our paid Splunk support answer CIFS is not supported for cold buckets from Linux/Unix. To quote Splunk:

As our documentation page stated that it is supporting CIFS as a data storage medium like frozen or archive directory. But you configured the cold DB to CIFS, as cold DB is part of the SPLUNK_DB working directory, therefore it is correct that the configuration is not supported. 

I find it disappointing that the Documentation states NFS can be used as "a storage medium" but CIFS can be used as "a data storage medium" - and the difference in the two is HUGE! (Eg. NFS for cold buckets is OK, CIFS is not). I believe this should be clarified.

0 Karma

jrodman
Splunk Employee
Splunk Employee

The reality is that both have awkward failure scenarios, but both should work on their native platforms. As you have encountered there is some lack of clarity on the precise boundary of what is supported here within the organization. Agreed this need to be cleared up.

halr9000
Motivator

I just edited the Answer above and added "from Linux/Unix", which makes @jrodman's above comment more explicit. I had someone ask me about this one today and this was not clear. Also, I just started a chat with the docs team who will be revising the linked pages and making everything clean.

As jrod says--native works. Unix can index to (insert caveats here) or roll cold to NFS, and Windows can do the same to SMB. But neither OS can hop across to the other protocol. These things could be made to work, but as nobody seems to want to do so, it's not been tested, so the current state is as described. Hope that makes sense.

0 Karma

jrodman
Splunk Employee
Splunk Employee

Last I checked, indexing to SMB/CIFS (hot / warm) did not actually work at all, even on Windows. It tried to work but would fail in various ways. I didn't dig deep enough to figure out whether this was a Splunk error or something else. Granted, this was around four years ago, so things may have changed.

0 Karma

halr9000
Motivator

Please open a support case, as this appears to be a bug in the product that needs swift resolution. If you don't have a support contract, get in touch and I will help escalate the issue appropriately.

Here's a docs link detailing support for CIFS.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...