Deployment Architecture

Introspection Replication

rsec39
Engager

I have a 2 peer cluster and have run into the following with 2nd cluster bundle deployment (version 6.1.2)

guid=3B88C1A4-84E0-4B8D-A6B4-4E6E5A5B71D8. index='_introspection' app='system' is configured for replication but not specified in bundle downloaded from master. Either define this index on master for replication; or specify repFactor=0 on peer to skip replication.;

Tags (2)

cstump_splunk
Splunk Employee
Splunk Employee

One more thing about this splunkd.log error message:

ERROR loader - Failed to download bundle from master, err="App='system' with replicated index='session_end' is neither in the bundle downloaded from master nor managed by local deployment client. Either define this index at the master or specify repFactor=0 on peer to skip replication. 

If you see this message on your search peer and it is causing the indexer to crash, the issue may be with locally defined indexes. If an index is created through the GUI, the index definition is saved in $SPLUNK_HOME/etc/system/local/indexes.conf. Clustering of the Indexers becomes a problem because of these local indexes.

An index that needs to be a clustered index needs to be in the master_apps directory on the Cluster Master. If there are local index definitions on the clustered indexer, remove them and make sure they are defined on the Cluster Master

mkolkebeck
Path Finder

By default, Splunk does not enable index replication (except for a handful of indexes, within master-apps/_cluster/default).

Configuring repFactor = auto by default on the cluster-master so that all indexes replicate was causing your same error for me, but not right away. The initial configuration bundle push was successful, but after any new configuration bundle pushes were made, Splunk complained of the same error.

[default]
repFactor = auto

The fix, I've explicitly set _introspection index with repFactor = 0, keeping the default to replicate all other indexes that are created.

[default]
repFactor = auto
[_introspection]
repFactor = 0

claudio_manig
Communicator

Hi - Thanks for the fix, this worked for me even setting it to "auto". But this seems to be a bug, was this reported to splunk support?

0 Karma

esalesapns2
Path Finder

I applied "repFactor = 0" for the _introspection index to an instance running SmartStore. If you do that, your Replication and Search factor will never be met, as shown on your Master Node "Indexer Clustering: Status" page.

0 Karma

mkolkebeck
Path Finder

I have not tested this - based on others' comments, it's possible the issue is likely not having setting repFactor explicitly defined for the _introspection index. Therefore, explicitly setting this index to repFactor = auto might be the best solution, especially if you [default]:repFactor = auto. And as @claudio.manig suggests, I recommend opening a support case with Splunk on this if you're continuing to have issues.

0 Karma

rkirkw
Path Finder

This helped me as well. With 2 indexer clusters, one started acting this way.

Simply added

 [_introspection]
 repFactor = auto

into my $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf and it fixed it.

These splunk bugs are really annoying and make my job twice as hard as it should be.

aholzer
Motivator

I recently upgraded from Splunk 6.0.3 to 6.1.3 and was receiving this error message.

The solution that finally worked for me was going to $SPLUNK_HOME/etc/master-apps/_cluster, making a copy of the default/indexes.conf into local/indexes.conf and adding the below lines at the end:

[_introspection]
repFactor = auto

Remember not to change the indexes.conf in your "default" folder. Always make a copy of any default configuration to local before changing it.

Hope this helps anyone else that encounters this problem.

0 Karma

nwales
Path Finder

Did you get this working?

It caused a major outage for me a few months ago, I had to edit the system/default/indexes.conf with repfactor=0 on the indexers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...