What do I need to do to ensure a clean Splunk migration?

[dharveynswccd]

In my environment I have an intermediate universal forwarder (syslog collector) which collects data from multiple sources and sends to this data to the indexers. We are deploying a new server and would like to know the following:
a. What directories need to be copied over to the new server after Splunk is installed.? Does it need to be the entire /opt/splunk directory and will that do the trick?
b. What do I need to do to ensure that the new server is checking in and sending logs to the search heads?

[mdsnmss]

Is this a duplicate syslog server to replace the other system? Are you using a deployment server to manage the intermediate UF? If you use one to remotely manage the forwarder you can add it to all the same server classes as the existing server. Otherwise, once Splunk is installed on the new system you should be able to just copy over /opt/splunk/etc. "Etc" is where your configs reside so as long as the directory structure on your new system is the same as the old server it should pick up on the inputs.

To check on your forwarder status you can check TCP connections and sending volume with this search:

index=_internal sourcetype=splunkd group=tcpin_connections (connectionType=cooked OR connectionType=cookedSSL) fwdType=* guid=* | eval dest_uri = host.":".destPort | stats values(fwdType) as forwarder_type, latest(version) as version, values(arch) as arch, dc(dest_uri) as dest_count, values(os) as os, max(_time) as last_connected, sum(kb) as new_sum_kb, sparkline(avg(tcp_KBps), 1m) as new_avg_tcp_kbps_sparkline, avg(tcp_KBps) as new_avg_tcp_kbps, avg(tcp_eps) as new_avg_tcp_eps by guid, hostname

You can just search for the specific host to see that it is checking in and sending data. It might also be a good idea to check what sourcetypes your forwarder is sending on the specific forwarder so you can validate they are all being received from the new server as well.

[dharveynswccd]

Thanks for the tip. Much appreciated