We have an interesting challenge before us. We have numerous firewalled enclaves that are also prone to frequent network outages. Due to the geographically dispersed and harsh environments (truly), it is possible for network drops to persist for days. Each enclave has a mix of legacy computers and ICS data sources.
These enclaves are necessarily firewalled from our "working LAN". The enclaves do connect to a "parent" DMZ though, and that network can communicate with the LAN. We brainstormed the following strategy, but would very much value some thoughts from the Splunk community. This is entirely new territory for us.
Our thoughts...
- Each enclave will have an indexer which will also serve as a search head. They will be sufficiently resourced.
- There will be 15 to 20 enclaves (site1, site2 ... site15, etc), each with its own unique group of data sources. Time before frozen will be set to 30 to 60 days (to tolerate network outages and have most current events searchable at the location).
- There will be a "parent" DMZ network which will also host a Splunk indexer and a search head, both of which will be well resourced. This will be site0
- We want to replicate data from the enclaves (site1, site2, and so on) to the site0 indexer. We do not think that this is going to exceed 50GB per day, in fact probably closer to 20GB daily.
- We use a LM provided to us by another "authority" but we do not have direct access to administrate it. We currently have a mature Splunk environment on our working LAN that connects to it. Unfortunately, we are not sure how to configure the proposed sites to the LM.
Technically, we don't yet know how to implement this strategy or if it is even feasible.
Questions we have for the Splunk community include:
Is this a feasible approach? Are there better strategies? We considered using UFs with ample storage but that does not solve the problems of having capacity for a multi-day outage (potentially a couple of weeks) nor making the data searchable within the enclave.
Would it be wiser (or necessary) to use a HF instead of an indexer at the enclave level to store and forward during outages, or is the indexer/site# doable?
Are we able to do this without clustering the sites?
How would be best handle connecting to the LM? Site0 will be able to connect to the LM. However, none of the enclaved sites will be able to reach out to the LM.
What would the one-way configuration even look like from sites1-x going to site0?
I realize that this is a lengthy post with lots of info and some questions but if you would have some thoughts or if you have similar experience that could be highly valuable to us and our efforts.
Thank you.
... View more