Hi, i saw many suggestions to routing data to different index from light forwarder but none seems to work. I have set up light forwarder that sends cooked data to my indexer. Connection is ok, because data gets indexed but in wrong index.
Here are some of configurations i have tried (on indexer side):
inputs.conf
[splunktcp://9997]
index = test_2
Next one:
inputs.conf
[splunktcp://9997]
props.conf
[host::Simpson-test]
index = test_2
TRANSFORMS-foo = routeIndex
transforms.conf
[routeIndex]
SOURCE_KEY = source
REGEX = = WinEventLog
DEST_KEY =_MetaData:Index
FORMAT = test_2
Next one:
inputs.conf
[splunktcp://9997]
sourcetype=REMOTE
props.conf
[REMOTE]
TRANSFORMS-simremote = sremote
transforms.conf
[sremote]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = test_2
Data allways ends up in index that was defined on lightforwarder (test_1). I would like to route data that comes to indexer based on sourcetype and host. Data comes from lightforwarder and is cooked so i suppose that sourcetype and host allready exist and i can do REGEX search on SOURCE_KEY?
Any hints abouth troubleshooting this?
The light forwarder can send to a specific index if you set the destination index in inputs.conf on the forwarder itself. This can be overridden at the indexer using props.conf/transforms.conf for a specific host or sourcetype.
props.conf:
[myfavoritesourcetype]
TRANSFORMS-index=sendtomyindex
transforms.conf:
[sendtomyindex]
SOURCE_KEY=MetaData:Sourcetype
REGEX=(.*)
FORMAT=index::myindex
WRITE_META=true
Update: it looks like this works just as well. I'm not sure which is preferable...
[sendtomyindex]
SOURCE_KEY=_MetaData:Index
DEST_KEY=_MetaData:Index
REGEX=(.*)
FORMAT=myindex
The easiest way is to specify the index in inputs.conf on the forwarder.
It seemed counter-intuitive to me at first, as if I was directing the forwarder to index something locally. But data is sent along to the indexer and placed into the desired index (as defined in an indexes.conf on the indexer).
On the forwarder, inputs.conf:
[monitor:///filename/or/other/input/type/]
disabled = false
index=ghtest5
The light forwarder can send to a specific index if you set the destination index in inputs.conf on the forwarder itself. This can be overridden at the indexer using props.conf/transforms.conf for a specific host or sourcetype.
props.conf:
[myfavoritesourcetype]
TRANSFORMS-index=sendtomyindex
transforms.conf:
[sendtomyindex]
SOURCE_KEY=MetaData:Sourcetype
REGEX=(.*)
FORMAT=index::myindex
WRITE_META=true
Update: it looks like this works just as well. I'm not sure which is preferable...
[sendtomyindex]
SOURCE_KEY=_MetaData:Index
DEST_KEY=_MetaData:Index
REGEX=(.*)
FORMAT=myindex
Side note on your regex: You should be able to use REGEX=.?
, which should be more efficient way to say match-all.
Interesting, I tried both of these (and some variations that didn't work) and only saw one "index" field in each case. But then again, I'm struggling to see "_index" in the UI. How are you seeing that?
Thanks alot, got it to work!
First one doesn't exactly work. It really adds new _index field to data, but doesn't delete old one. So I end up having two _index fields, but only first _index field is used to store data, and second one seems to get ignored.
Second one works great if i write it like this:
[routeIndex]
SOURCE_KEY = MetaData:Sourcetype
REGEX=(.*)
DEST_KEY=_MetaData:Index
FORMAT=test_2
WRITE_META=true