Getting Data In

Have forwarder duplicating data to 2 indexes.

mataharry
Communicator

For some inputs on a forwarder, I want to send the same data to the same indexer, but duplicate it in 2 indexes (they have different permissions/retention)

This is sending to the main index, I want to main and public.
[monitor:///var/log/feed]
disabled = false
followTail = 0
sourcetype = one
index=main

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

2 methods :
A - Use another instance of splunk monitoring the same file and specifying a different index
(by example on windows you can have 1 UF and 1 LMF/HF/indexer, on linux, as many instances as you want)

B - use a symlink to the files/folders and have a secondary monitor on the symlink (with a different index destination)
see screenshot for the result

create a symlink
example :
ln -s feed symlink

and define 2 inputs one on the original, the other on the symlink

[monitor:///var/log/feed] 
disabled = false 
followTail = 0 
sourcetype = one 
index=main 

[monitor:///var/log/symlink] 
disabled = false 
followTail = 0 
sourcetype = one
 # or any other sourcetype 
crcSalt=< SOURCE >
 # required to force splunk to differentiate files based on the path/filename, write SOURCE in caps (the html formatting may hide it), and remoce the space in the tag.
index=public
 # the index of your choice 
followSymlink=true 
 # to make sure that the symlink will be followed. 

View solution in original post

yannK
Splunk Employee
Splunk Employee

2 methods :
A - Use another instance of splunk monitoring the same file and specifying a different index
(by example on windows you can have 1 UF and 1 LMF/HF/indexer, on linux, as many instances as you want)

B - use a symlink to the files/folders and have a secondary monitor on the symlink (with a different index destination)
see screenshot for the result

create a symlink
example :
ln -s feed symlink

and define 2 inputs one on the original, the other on the symlink

[monitor:///var/log/feed] 
disabled = false 
followTail = 0 
sourcetype = one 
index=main 

[monitor:///var/log/symlink] 
disabled = false 
followTail = 0 
sourcetype = one
 # or any other sourcetype 
crcSalt=< SOURCE >
 # required to force splunk to differentiate files based on the path/filename, write SOURCE in caps (the html formatting may hide it), and remoce the space in the tag.
index=public
 # the index of your choice 
followSymlink=true 
 # to make sure that the symlink will be followed. 

mataharry
Communicator

I tested with multiple groups in outputs.conf but I cannot change the index and they all go to the same index.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...