What are best practices with creating indexes and sourcetypes for two distinct sources of unstructured data from the same application?

[markwymer]

Hi all,

I couldn't find any definitive answers, so I'm hoping that the forum members' real life experiences may point me in the right direction.

We have lots of logged events (over 300 million per day) from two, distinct sources for the same application. Even though the event message format is different between the two types of source data, there is one major commonality - the raw events are completely unstructured!

When we first got Splunk, we only had one data source which made it simple to set up an Index and a Custom Sourcetype.

When we started ingesting the second source, there was a huge pressure to get the data in and analysed which left me little time to appraise the actual raw data. In my rush ( and, perhaps, naivety!!) I created a second index and a second sourcetype and, as a result of this, many of our searches start with index=abc OR index=xyz etc!

My question is, given the volume of data and the similarity in that it's all unstructured data, would it be best, from an indexing and searching perspective, to move towards one of:
1 index and 2 sourcetypes
2 indexes and 1 sourcetype
1 index and 1 sourcetype
Or stick with our current
2 indexes and 2 sourcetypes
?

Thanks for any ideas,
Mark.

[ddrillic · Accepted solution]

Mark, what you have been going through seems to be normal. My take is that at some points in the evolution of the Splunk usage, we need to re-think the design at the content level as well as the infrastructure level.

The sourcetype is always an interesting choice. A good discussion is at - sourcetype best practices

[sundareshr]

IMHO, a lot of this depends on your usage. Having said that, here are some pointers to consider

1) Retention: Will both data sources have same retention policy for warm, cold, frozen. archived etc.?
2) Permission: Does everyone need to have access to data from both sources. Its easier to set access by index.
3) Performance: mixed data results in larger number of keywords=larger index.
4) Data volume: Create separate index for high volume data sources.

In your case, since you already have the data in two indexes, I would just leave it alone.