How to store logs in WORM disk?

[ds1405s]

Hi,

Is it possible to use WORM disks as a storage for splunk indexer?
Otherwise, is there any possible ways for stored logs to prevent from alteration & forgery?

Thanks,
HWKim

[stevepetr]

The easiest way to prevent any data, including log files, from alteration and forgery is to place them on WORMdisks which are SATA hard disk drives with Write Once Read Many (WORM) capability for permanent protection (see https://en.wikipedia.org/wiki/Write_once_read_many) and (https://greentec-usa.com). These WORMdisks appear as an ordinary hard disk drive to the system and use standard SATA, USB, or eSATA and work with standard applications (e.g. use as the 😧 drive or mount point). This way the log files can be protected online and remain searchable. They can be used as individual disks or networked NAS or SAN storage. This is a NIST SP 1800-11 Data Integrity standard technology to be tamper-proof and impossible to circumvent.

[dwaddle · Accepted solution]

It may be possible, with some restrictions, to use WORM disks for Splunk storage. However, it may not be practical. One big technical restriction is that Splunk's HOT buckets could not be stored on WORM storage, as the action of indexing requires many rewrites of the index data itself. Once a bucket is rolled to COLD, however, it is no longer written to and it would be possible to store it on WORM media.

However, most WORM media is either tape or optical - neither is exactly conducive to "painless" normal filesystem operations. You could look at some (very specialized) equipment like a UDO optical library ( http://www.plasmon.com/archive_solutions/glibrary.html ) using software like AMASS ( http://www.quantum.com/Products/Software/AMASS/Index.aspx ) to make the optical library appear as a large POSIX filesystem.

None of the above qualifies as either simple or cost effective. And, Splunk themselves may not support it without a LOT of qualification testing.

Another option that just came to me is "software" WORM using something like NetApp's Snaplock Compliance ( http://www.netapp.com/us/products/protection-software/snaplock.html ). Snaplock should allow you to treat a chunk of NFS storage as effectively write-only. Again, this is only good for Splunk COLD storage, because HOT buckets are frequently rewritten as new data arrives and there is no way (as of 4.3) to separate HOT and WARM data onto different filesystems. ( There's also the side comment that Splunk doesn't recommend use of NFS for hot buckets. ) So, you could wind up with a solution where once logs reach a certain age (and get rolled from hot/warm -> cold) that they are effectively tamper-proof. ( Assuming, of course, the NetApp Snaplock software is 100% reliable. ) Some issues may remain though, in areas like how do you expire aged logs from the write-only appliance? (Which is not only not over-writable, but not deletable) Also, there is the time period where data must be in a hot bucket that is fully rewritable. During that time, without additional controls, tampering is possible.

Before trudging down this path, I would verify the exact nature of the requirement. And I would look at Splunk's built-in options like IT data block signing ( http://docs.splunk.com/Documentation/Splunk/latest/Admin/ITDataSigning ) to see if simpler approaches can solve this requirement.

And, as with any solution designed to meet a regulatory requirement or solution that includes a cryptography component - it's best to discuss with experts who have studied the problem in detail. Splunk's professional services organization would be best suited to provide consultative support in this area and know exactly what the product can/cannot do, leaning on their experiences solving this problem in the past in a variety of situations.

[ds1405s]

Thanks again,

It could be better that splunk's built-in option has a hash-chain mechanism for tamper-proof/tamper-evidence.

I'll try several combinations that you suggested and find out what is the best answer(at now) for the requirements.

Regards,
HWKim

[dwaddle]

Note: Frozen != Cold. Frozen is not searchable, cold is. I've also updated the answer w/ more info. Also, there's a subtle difference between "preventing" something and making that thing "impossible". Compared to plain text logs, indexing your logs in Splunk raises the bar against tampering. Signing raises that bar further. WORM is another level of prevention, but still doesn't make tampering impossible... in the end, it is a judgement call between cost vs risk. A line printer in a vault w/ 2 armed guard(s) may be the only 100% impossible tamper-proof solution.

[ds1405s]

Thanks for your answers, dwaddle & clyde.

The reason why I asked those questions was,
almost every RFPs releated to financial/government require that logs must be prevented from alteration and forgery (And searchable!!).

Storing logs to WORM-like media might be the simplest solution.
But, Splunk's frozen buckets doesn't look searchable.

Digital signing could be alternative but there will be some other issues such as algorithms, certified cryptographic Module(CMVP), verification loads, etc. (Self-signed certificate is not be acceptable)

Regards,

[clyde772]

김 차장님,

위의 내용처럼 확인해봐야할 일들이 많은듯 합니다. 무선 정확한건, COLD가 되면 데이터에 update는 더이상 발생되지 않는다고 볼때, COLD 저장 경로의 위치를 WORM으로 하면 될것 입니다.