Splunk Search

Sending Single Input To Multiple Indexes: Works Sometimes

jpass
Contributor
  • I have a scripted input with events that I want to send to different indexes based on a string within the event.
  • I do this for easier access control.
  • I control the event format via the script and so have created an easy-to-regex string at the bottom of each event which Splunk should use to route the event.
  • It works but only sometimes. I'll import 10 events and some go to the default index and some go to the routing index.
  • These are multi-line events that are usually about 20 to 50 lines but sometimes up to a few hundred lines (rare)
  • I'm wondering what I'm doing wrong.

Here's my .conf entries

inputs.conf (in a custom application directory /splunk_home/etc/apps/myapp/local/inputs.conf)

[script:///splunk_home/bin/scripts/get_test_multi.pl]
disabled = 0
interval = 60000
source = get_test_multi
sourcetype = test_multi
index=test_index

props.conf (in /splunk_home/etc/system/local/props.conf)

[test_multi]
TRANSFORMS-index=test_index_router
EXTRACT-test_checker = splunk_index :: (?<test_index_field>.*?)\n

transforms.conf (in /splunk_home/etc/system/local/transforms.conf)

[test_index_router]
REGEX = splunk_index :: (?<my_splunk_index>.*?)\n
FORMAT = $1
DEST_KEY = _MetaData:Index

In props.conf you'll notice I extract a field called 'test_index_field' which is the same as the regex I'm using to route the index. I do this just to confirm that I'm correctly extracting the value which works correctly 100% of the time.

  • The events are sent to the correct index only some of the time
  • Events are either routed correctly or sent to the value I designate in inputs.conf for index
  • I've tried removing the entry for index in inputs.conf with the same results only it sends the events to index=main (default)

jacobwilkins
Communicator

You might have some whitespace issues with your regex.

Try this:

REGEX = splunk_index :: (\w*)

The concern is that you may be including training whitespace in the capture of your index name, which would be tricksy to debug.

You might also want to try this:

[source::get_test_multi]
TRANSFORMS-index=test_index_router
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...