Restructuring Splunk data index best practices?

[JordanPeterson]

I inherited a rather messy Splunk deployment and within the next 2-6 months will be starting a project to reorganize our deployment to be more understandable for the next poor soul to inherit it.

Right now my focus is on fixing our indexes. We have a lot of data going to a lot of indexes and nobody seems to quite understand why what is going where. We are looking for resources and suggestions on Splunk best practices when it comes to indexes and how to separate data. I think I would like to be able to separate data based on type (Operating System, Network, Application, etc.) and there is also a data privacy concern where we need to separate data based on access (we don't want developers seeing customer info etc).

If you had the opportunity to completely revitalize your Splunk deployment where do you recommend getting started and how would you organize your indexes?

[gcusello]

Hi JordanPeterson,
there are two main rules to organize your indexes structure:

• access rights,
• retention period.

In addition it's a best practice to distribute logs in more indexes to have more performaces.
At the same time it's better to not grow the indexes number because there's the risk to lose indexes management.

At the end I use to have at least an index for each App or environment I'm monitoring.

So I suggest to follow the first two main rules and at the same time to have a medium number of indexes.
Use always a naming rule to understand application area.

At the end, before starting you reorganization: define rules writing a document and design on paper your architecture.

It's very important:

• for the future for next requirements,
• for the immediate period to modify indexes addressing of your Apps and TAs.

I hope to be helpful.

Bye.
Giuseppe