Forward to other index

[Koboldus]

Dear all,

may I ask a noob-question to the experts?

Currently I am forwarding Data from several forwarders (F_a, F_b, F_c) to a splunk indexer (S_a). So like this:
F_a
F_b --- > S_a (These are collected in 3 different Indexes: a, b, c)
F_c

for Research purposes I would now use all the data that is sent to S_a in another indexer (S_b). So like this

F_a
F_b ---- > S_a ---- > S_b
F_c

This can be done very easy of course by using the "Configuring Forwarding" in the mangement console. The challenge what I have is, that I want all the data comming from S_a to S_b to be collected in one single index, e.g. "abc". So in Terms of Indexes it is like this:

Indexes on S_a:
a
b
c

Index on S_b:
abc

The idea is then to feed all data from Indexes a,b,c (from S_a) to the single index abc (in S_b). And I would like to have that not "one time" but real time during forwarding.

Is that possible and how? In an ideal case, the Information about the original index can be kept in an additional field then.

best regards and thanks a lot for your answers in advance

[Koboldus]

Thanks to all of you for the answers.

The Problem I encountered with the solution provided before was, that I had to put the transforms.conf and props.conf on s_A and not on s_B

that leads to the index-Transformation on s_B.

Problem solved!
thanks again and best regards

[mwdbhyat]

This doc will probably help you http://docs.splunk.com/Documentation/Splunk/4.3.3/Deploy/Routeandfilterdatad

[gcusello]

Hi Koboldus,
if you want to override index name in Indexer B see at https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides
In other words, on Indexer B you have to insert
transforms.conf

 [overrideindex]
 DEST_KEY =_MetaData:Index
 REGEX = .
 FORMAT = my_new_index

props.conf

 [mysourcetype1]
 TRANSFORMS-index = overrideindex
 [mysourcetype2]
 TRANSFORMS-index = overrideindex
 [mysourcetype3]
 TRANSFORMS-index = overrideindex

Bye.
Giuseppe

[Koboldus]

To add more details to it:
I tried out the concept by manipulating the sourcetype like this:

props.conf:
[host::MYHOSTNAME]
TRANSFORMS-index = overrideindex

transforms.conf:
[overrideindex]
REGEX = .
FORMAT = sourcetype::my_log
DEST_KEY = _MetaData:Sourcetype

I put both files in
$SPLUNK$/etc/system/local/

but when the S_a is sending data again, it stays the same as it was on the sending indexer. I would assume that the sourcetype had changed to "my_log".

best regards

[Koboldus]

To add more details to it:
I tried out the concept by manipulating the sourcetype like this:

props.conf:
[host::MYHOSTNAME]
TRANSFORMS-index = overrideindex

transforms.conf:
[overrideindex]
REGEX = .
FORMAT = sourcetype::my_log
DEST_KEY = _MetaData:Sourcetype

I put both files in
$SPLUNK$/etc/system/local/

but when the S_a is sending data again, it stays the same as it was on the sending indexer. I would assume that the sourcetype had changed to "my_log".

best regards

[Koboldus]

Thanks a lot so far. I tried many different variations of the solution proposed by Giuseppe.

Selecting Events by sourcetype or by host
and changing to another index as well as modifiying sourcetype or host
Regex only with . or with parantheses (,) or with (.*)
(I even double checked for typos with a colleague)

but on the sending machine (S_a in the original post) I receive then the following error message in splunkd.log:

TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 70 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Could you give advice, how I can figure out the cause of this warning? Are there more sophisticated logs where the error I made is pointed our more in Detail?

best regards

[niketn]

@Koboldus, refer to Splunk Documentation on the collect command which you can use to move data from different indexes to a single index:

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Collect#Moving_events_to_a_diffe...

[lfedak_splunk]

Hey @koboldus! Welcome to the Splunk community. We also have a noob channel in our Slack community. You can check out this link if you'd like to join it! http://splk.it/slack
If these users answered your question, please remember to accept an answer to award karma points. Happy Splunking!