Lost all indexes

[satiex]

Hi,

I created an App on my Splunk indexer which had a props.conf, outputs.conf and transforms.conf with the following data:

[syslog]
TRANSFORMS-routing = routeAll, routeSubset

Edit $SPLUNK_HOME/etc/system/local/transforms.conf and add the following text:

[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything

[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything

Edit $SPLUNK_HOME/etc/system/local/outputs.conf and add the following text:

[tcpout]
defaultGroup=nothing

[tcpout:Everything]
disabled=false
server=10.1.12.1:9997

[tcpout:Subsidiary]
disabled=false
sendCookedData=false
server=10.1.12.2:1234

I then restarted my Splunk service and it froze while restarting. It then appeared that my server had frozen, so I restarted the server on the hypervisor level after waiting some time with splunk not coming back online.

Once it was back online, I realised that all Indexes were completely wiped from the mount point, and the indexes reset and started populating with new data coming from the heavy forwarder.

Is there any way that changes to conf files could have done this? Or could this have happened if the server crashed while Splunk was restarting?

Thank you.

[adonio · Accepted solution]

outputs.conf on the indexers is a really bad idea ...
you literally tell the indexer to ship its data out

[satiex]

I see. It sounds like that is what happened.

What I'm trying to accomplish is "Replicate a subset of data to a third-party system" from this page: http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad

HF recieves data via syslog from a source. I want to filter the syslog from that source and send a subset of that to another syslog server.

I will create a new ticket to ask about how to achieve this, as the example given is for forwarding raw data only from the HF. It says that if you want to forward syslog data it needs to be output from an indexer.