Index strategy? Single index or multiple indexes?

[archananaveen]

Hi there,

I have around 80 servers and about 300 folders in all of them. Should we use a single index or use a multuple indexes while adding every folder as sourcetype? I also want to give access to users for these folders, how could we control if we end up using single index with LDAP authentication?

Anyways to do that using specific dashboards with LDAP Auth?

Regards,

Reference link:
https://answers.splunk.com/answers/459071/index-strategy-single-index-with-multiple-sourcety.html#an...

[archananaveen]

Thank you for the reply.

Where could I find more infor on controlling access for a single index setup at the GUI level?

If a data is alredy captured /a/b/* and in the future if I add a new additional index for /a/b/c, what are the consequences? Will there be duplicate information?

If we decide to have 300 indexes, will it affect the search time? Or any process contention for the various forwarders?

[s2_splunk]

Not sure what you mean by

controlling access for a single index setup at the GUI level

If you use a single index, you cannot have different permissions for various roles. I must be misunderstanding.

Splunk doesn't duplicate information. Why would you add another index for the same sourcetype in the future, what's the expected use case?

The number of indexes only affects your search performance if you do searches like index=*. You should always be explicit which index (or indices) you are searching in. Also, forwarders don't have any relationship with number of indices. Every source the forwarder monitors is configured to be indexed in a single index. The rest is handled by the indexer; nothing to consider here really.

[archananaveen]

ok, if single index data is for the support team supporting various applications. In the future we might be asked to display app related infomration to clients. Since we won't be able to give them access to single index(regulatory issue) we might as well add a new separate index for /a/b/c in that case. Does that make sense? How else could we accomplish such a request?

[s2_splunk]

You can either plan ahead and create app-related indices for the get go, or you do it later. If you decide to do it later, your support team would have to include the new indices in their search queries.

So, if they search index=allInOne today, they would need to do index=allInOne OR index=app in the future. You can abstract all of that away by either using the sourcetype in search or by creating eventtypes or macros. Then you have to only change it in one place later on.

[archananaveen]

Sounds Good, Thank you!

[archananaveen]

Soungs Good, Thank you!

[s2_splunk]

There are three factors that require data separation into multiple indices:

1. Access Permissions are different
2. Data retention requirements are different
3. Data comes in at vastly different velocity/volume
   

While you can protect access via apps as well, you need to ensure that users have no other way of running searches outside of the app context. I would not rely on that and apply access permissions at the index level for sure.
I would think about which user groups should be able to see what data and structure indices and RBAC around that. Keep it simple.

BTW: The sourcetype should describe what the data really is, e.g. web_server_logs, firewall, appXyz, etc. You will already have the folder name in the source metadata field.

Hope that helps.