Deployment Architecture

Can i single file be forwarder to 2 different indexes ??

rakesh_498115
Motivator

Hi Team.

I have a file name called /var/log/messages which I need to send to two different indexes (say index="A" and index="B") on 2 set of indexer servers.

i.e index="A" sourcetype="A" on the indexers set 1 - and index="B" sourcetype="B" on the indexers set 2 -- should give me the same source list i.e /var/log/messages/

please help

thanks.
rakesh.

Tags (1)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

I believe it would involve two steps, data cloning and metadata override. Before we go further, let me remind you that it would cause double license usage for the file.

Assuming you've a data input (inputs.conf) setup to monitor the file with assigned index=indexA and sourcetype=sourcetypeA.

Step 1: Data cloning
You could configure your forwarder to clone the same data to two different indexers. See below link on corresponding outputs.conf configuration

http://docs.splunk.com/Documentation/Forwarder/6.5.0/Forwarder/Configureforwardingwithoutputs.conf#C...

Step 2: Metadata override
On Indexer two have following props.conf and transforms.conf

props.conf

[sourcetypeA]
...other linebreaking/timestamp recognition settings..
TRANSFORMS-override = changeindex,changesourcetype

transforms.conf

[changesourcetype]
FORMAT = sourcetype::sourcetypeB
DEST_KEY = MetaData:Sourcetype
REGEX = .

[changeindex]
DEST_KEY =_MetaData:Index
REGEX = .
 FORMAT = indexB
0 Karma

vr2312
Contributor

That should work if i am not wrong. Cause we do have a lot of issues where there is duplicacy in the logs getting indexed.

I would recommend you to try it out on a test environment to be cautious.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...