Discussion: Practical cases of going against Splunk Best Practices

[j4adam]

Hello all,

Potentially a bit of a sensitive topic, but I wanted to see what others thought.

Splunk Best Practice are great and really help installations to go smoothly and work optimally, but I can think of at least one case where it's not always practical to follow them.

My example is something I have done on all of my ES deployments: Install DBX on the ES SH when needed (best practice is to have no additional apps installed on the ES SH). I do this because some environments use DBX to collect asset data and, while you could index it, it's much simpler to just write directly to a CSV using a scheduled search.

Asset data is a type of data where (when using a well made search) the old data is of not actionable value because the newest data should be a complete picture of your environment, so installing DBX on a forwarder and indexing it is a waste of storage paste (regardless of how small) and adds additional complexity that does not need to be there.

I understand the reasoning behind "no additional apps on the ES SH" is to prevent bloat and take precious resources away from a very hungry system, but I treat this best practice as a rule of thumb that should be approached at a case by case basis .Having a single search run at 1 AM every day is going to have exactly 0 performance impact, and if it does you've got bigger problems.

I've never had any issues doing this, until recently were someone was told to remove DBX from the ES SH because it wasn't a best practice, which caused a few headaches and, in my opinion, caused more problems by fixing an issue that didn't exist.

---

What are your thoughts on this? Do you have any other examples of best practices being a great guideline, but not a rule of law?

[mosman_splunk]

Splunk recommend not installing greedy apps, I mean apps with a lot of saved searches, data models acceleration, and eventtypes. check the skkiped searches and average load time, if your environment is stable engought, I don't see a reason why not.

[dslife_splunk]

My personal opinion is Best Practices are just that. Good things to do in most cases, but there are times where a system owner will decide that because of a specific requirement to do it differently.

However, for your asset tracking use case if the external system doesn't maintain history of old assets, or older information about current assets you might want to indexing it so you can link it with old data in a historical security investigation.