Getting Data In

Have forwarder duplicating data to 2 indexes.

mataharry
Communicator

For some inputs on a forwarder, I want to send the same data to the same indexer, but duplicate it in 2 indexes (they have different permissions/retention)

This is sending to the main index, I want to main and public.
[monitor:///var/log/feed]
disabled = false
followTail = 0
sourcetype = one
index=main

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

2 methods :
A - Use another instance of splunk monitoring the same file and specifying a different index
(by example on windows you can have 1 UF and 1 LMF/HF/indexer, on linux, as many instances as you want)

B - use a symlink to the files/folders and have a secondary monitor on the symlink (with a different index destination)
see screenshot for the result

create a symlink
example :
ln -s feed symlink

and define 2 inputs one on the original, the other on the symlink

[monitor:///var/log/feed] 
disabled = false 
followTail = 0 
sourcetype = one 
index=main 

[monitor:///var/log/symlink] 
disabled = false 
followTail = 0 
sourcetype = one
 # or any other sourcetype 
crcSalt=< SOURCE >
 # required to force splunk to differentiate files based on the path/filename, write SOURCE in caps (the html formatting may hide it), and remoce the space in the tag.
index=public
 # the index of your choice 
followSymlink=true 
 # to make sure that the symlink will be followed. 

View solution in original post

yannK
Splunk Employee
Splunk Employee

2 methods :
A - Use another instance of splunk monitoring the same file and specifying a different index
(by example on windows you can have 1 UF and 1 LMF/HF/indexer, on linux, as many instances as you want)

B - use a symlink to the files/folders and have a secondary monitor on the symlink (with a different index destination)
see screenshot for the result

create a symlink
example :
ln -s feed symlink

and define 2 inputs one on the original, the other on the symlink

[monitor:///var/log/feed] 
disabled = false 
followTail = 0 
sourcetype = one 
index=main 

[monitor:///var/log/symlink] 
disabled = false 
followTail = 0 
sourcetype = one
 # or any other sourcetype 
crcSalt=< SOURCE >
 # required to force splunk to differentiate files based on the path/filename, write SOURCE in caps (the html formatting may hide it), and remoce the space in the tag.
index=public
 # the index of your choice 
followSymlink=true 
 # to make sure that the symlink will be followed. 

mataharry
Communicator

I tested with multiple groups in outputs.conf but I cannot change the index and they all go to the same index.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...