Best practice for forwarding data to a differently named index?

[msellery]

I have three independent geographic sites, A, B, C.

A forth site, Z, needs a searchable copy of all data from A, B, C. Because sites A, B, C also need to search their own data locally, they forward to Z with IndexAndForward=true. This works great.

On Site Z, though, I need some way to flag which site the data came from. I thought about creating INDEXA, INDEXB, INDEXC on Site Z, so that I could add "index=INDEXC" to searches done on Site Z. However, I don't know how to forward data to a specific Index name without also changing that index name at the source of A, B, C. They all just use "Main" right now as a default index, which will cause other problems if it needs to change.

Can we store in "Main" locally but forward to a differently named Index on Z? What would the best practice be? Thanks!

[msellery]

Appreciate all the input here! It is a bit of a strange situation, but it's driven by data ownership politics. Sites A, B, and C need to function completely independently, including when disconnected from Site Z. But the people at Site Z are entitled to see and search all the data from all sites, and must be able to do independently even if disconnected from A, B, or C. As best I know, this may mean ingesting the same data twice, and paying the license accordingly.

I have not looked at multi-site index clusters so I'll have to read about that. Hopefully it doesn't involve data replication. For example, data at site B should never touch hardware at site A. Data from A, B, C should only be stored in the same physical location if that location is site Z.

Please keep the tips coming!

[adonio]

seems like your requirements makes the multi-site cluster even more viable,
multi-site clustering does involve replication, however, you can set the replication in any fashion you like.
for your use case:
Site A - 1 copy searchable and replica and sends replication only to site Z
Site B - 1 copy searchable and replica and sends replication only to site Z
Site C - 1 copy searchable and replica and sends replication only to site Z
now you have all data in site Z available for search regardless. and data will never replicated between sites A, B and C.

cheers

[msellery]

Adonio, that's exactly what I need! So it sounds like I need to learn about how to set all that up. And is it true that with multi-site I won't have to pay a double license ingestion penalty indexing at site Z as well? Also, do I still need to manage index names or will Multi-site give me an option for easily determining where data originally came from? Thanks!

[adonio]

hi @msellery,
It is true, you will not need to pay additional license ingestion.
you will not have to manage indexes names at all
you will have for example data as follow:
site A - index 1, index 2, index 3
Site B - index 1, index 2, index 4
Site C - index 1, index 3, index 5
Site Z - index 1, index 2, index 3, index 4, index 5
but you will have one set of indexes configurations across all sites and indexers if you wish to
start slowly by reading the relevant docs, link in my first comment, and keep us posted here as how it goes and if you have any further questions.
good luck

[koshyk]

Personally, I feel what you are trying to do may become complex later. This is because in SiteA,B,C etc, your use-cases or saved-searches may be running specific to an index or sourcetype etc. When you change it in siteZ, this may break.

Option1: Is there any chance for you do a Storage level or OS level synchronisation to SiteZ? Then virtualise your SiteZ accordingly and have your indexes cluster names as SiteA_cluster , SiteB_cluster. This can save you indexing license cost.

Option2: To use different Indexer instances on SiteZ. For example, install 3 instances in SiteZ (ensure your hardware have enough power) and forward from SiteA to Instance1, SiteB to instance2 etc. Then Run Search Head on all these 3 instances as multiple index cluster

[adonio]

very interesting way of moving your data around.
have you considered multi site indexer clustering?
read here:
https://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Multisiteclusters
might be a better and easier option.
also scalable down the road if you would like to add more geographies
hope it helps

[somesoni2]

It'll also help you save on licensing as right now it would be causing you double license usage with indexAndForward option. If you have to implement it without clustering, you can create indexA in siteA, indexB in siteB..and so on for your data, so when you forward it to siteZ, they'll already be in separate indexes.