Our Splunk instance is currently receiving data from a remote Splunk instance. The remote indexer is sending data (many hosts with many different sourcetypes) to our indexers over TCP port 9998. We are interested in forcing this data to be collected in a custom index.
I have confirmed that we are receiving data from the remote Splunk on port 9998, however, it is not being collected in the desired index. The following are the inputs.conf, props.conf, and transforms.conf which I currently have in place:
inputs.conf
[splunktcp://:9998]
props.conf
[source::tcp:9998]
transforms.conf
[setIndexMeta]
DEFAULT_VALUE = unknown
REGEX = (.)
DEST_KEY = _MetaData:Index
I would appreciate assistance with this.
"When you forward structured data to an indexer, Splunk Enterprise does not parse this data once it arrives at the indexer, even if you have configured props.conf on that indexer with INDEXED_EXTRACTIONS. Forwarded data skips the following queues on the indexer, which precludes any parsing of that data on the indexer:
parsing
aggregation
typing
The forwarded data must arrive at the indexer already parsed."
http://docs.splunk.com/Documentation/Splunk/6.1.4/Forwarding/Routeandfilterdatad
An expensive work around could be done by adding this to the inputs.conf
[splunktcp://9998]
route=has_key:_utf8:parsingQueue;has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue
Found here:
http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible...
http://answers.splunk.com/answers/5528/forwarding-select-data-in-my-environment.html
Forget props.conf
and transforms.conf
and just do this inside $SPLUNK_HOME/etc/apps/MyApp/default/inputs.conf
:
[splunktcp://:9998]
index=CustomIndex
sourcetype=MySourceType
You also need to make sure you check/add this inside $SPLUNK_HOME/etc/system/local/default-mode.conf
(it defaults to disabled):
[pipeline:tcp]
disabled = false
Then you need to bounce all Splunk instances on the servers that get these files.
In addition: The lines in default-mode.conf
are only needed if this will be done on a forwarder.
I thought this was because the intermediate indexer sending cooked data to the final indexer? If so I was thinking that using the route settings as described in this answer would make sure the data goes though the parsing queues again. Does setting you mention for default-mode.conf do something similar? Thanks..
What do you mean by "intermediate indexer"? What is your architecture?
Sounds like from the original poster that they are receiving data from another indexer. Thought the data might be fully cooked by the time it gets to him and not go though the parsing queues to set the the new index.
"The remote indexer is sending data to our indexers over TCP port 9998"
It sounds like the OP has a remote Splunk instance and is forwarding data from that instance to his main instance. He wants to force the data coming from the remote instance into a certain "special" index and not into the main/default/or whatever index the remote instance is putting the data into.
If parsing is already done by something else then the indexer is going to ignore the props and transforms, so I see why you said that they can be forgotten. But he has the index in inputs and it seems that it still isn't working.