Getting Data In

Why is data received from a remote Splunk instance not being collected in the specified index?

adamblock2
Path Finder

Our Splunk instance is currently receiving data from a remote Splunk instance. The remote indexer is sending data (many hosts with many different sourcetypes) to our indexers over TCP port 9998. We are interested in forcing this data to be collected in a custom index.

I have confirmed that we are receiving data from the remote Splunk on port 9998, however, it is not being collected in the desired index. The following are the inputs.conf, props.conf, and transforms.conf which I currently have in place:

inputs.conf

[splunktcp://:9998]

index=CustomIndex

props.conf

[source::tcp:9998]

TRANSFORMS-force_index = setIndexMeta

transforms.conf

[setIndexMeta]
DEFAULT_VALUE = unknown
REGEX = (.)
DEST_KEY = _MetaData:Index

FORMAT = CustomIndex

I would appreciate assistance with this.

0 Karma

jclehmuth
Path Finder

"When you forward structured data to an indexer, Splunk Enterprise does not parse this data once it arrives at the indexer, even if you have configured props.conf on that indexer with INDEXED_EXTRACTIONS. Forwarded data skips the following queues on the indexer, which precludes any parsing of that data on the indexer:

parsing
aggregation
typing
The forwarded data must arrive at the indexer already parsed."
http://docs.splunk.com/Documentation/Splunk/6.1.4/Forwarding/Routeandfilterdatad

An expensive work around could be done by adding this to the inputs.conf

[splunktcp://9998]
route=has_key:_utf8:parsingQueue;has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

Found here:
http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible...
http://answers.splunk.com/answers/5528/forwarding-select-data-in-my-environment.html

0 Karma

woodcock
Esteemed Legend

Forget props.conf and transforms.conf and just do this inside $SPLUNK_HOME/etc/apps/MyApp/default/inputs.conf:

[splunktcp://:9998]
index=CustomIndex
sourcetype=MySourceType

You also need to make sure you check/add this inside $SPLUNK_HOME/etc/system/local/default-mode.conf (it defaults to disabled):

[pipeline:tcp]
disabled = false

Then you need to bounce all Splunk instances on the servers that get these files.

MuS
SplunkTrust
SplunkTrust

In addition: The lines in default-mode.conf are only needed if this will be done on a forwarder.

0 Karma

cramasta
Builder

I thought this was because the intermediate indexer sending cooked data to the final indexer? If so I was thinking that using the route settings as described in this answer would make sure the data goes though the parsing queues again. Does setting you mention for default-mode.conf do something similar? Thanks..

0 Karma

woodcock
Esteemed Legend

What do you mean by "intermediate indexer"? What is your architecture?

0 Karma

cramasta
Builder

Sounds like from the original poster that they are receiving data from another indexer. Thought the data might be fully cooked by the time it gets to him and not go though the parsing queues to set the the new index.

"The remote indexer is sending data to our indexers over TCP port 9998"

0 Karma

jclehmuth
Path Finder

It sounds like the OP has a remote Splunk instance and is forwarding data from that instance to his main instance. He wants to force the data coming from the remote instance into a certain "special" index and not into the main/default/or whatever index the remote instance is putting the data into.

If parsing is already done by something else then the indexer is going to ignore the props and transforms, so I see why you said that they can be forgotten. But he has the index in inputs and it seems that it still isn't working.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...