Getting Data In

How to route data to certain indexes based on host, sourcetype, and index?

mahesh_ravji1
Explorer

Hi.

I have a requirement to route events to index based on the fields host, sourcetype, and index.

Field host format is dev-customerA, dev-customerB, etc
Field sourcetype is typeA, typeB, etc

The following routing rules are required:
- If event index is NOT 'main' then don't do any routing (i.e. let the event go to the index set in the event).
- Set index to customer part from host field (e.g. customerA, customerB, etc)
- For sourcetype = typeA and typeB, append '-keep' to the index (e.g. the index becomes customerA-keep, customerB-keep, etc)

Examples:
Event1 index=firewall host=dev-customerA sourcetype=ASA. Should not be routed as index does not equal 'main'
Event2 index=main host=dev-customerA sourcetype=ASA. Should be routed to index=customerA
Event3 index=main host=dev-customerA sourcetype=typeA. Should be routed to index=customerA-keep
Event4 index=main host=dev-customerA sourcetype=typeB. Should be routed to index=customerA-keep
Event5 index=main host=dev-customerB sourcetype=ASA. Should be routed to index=customerB
Event6 index=main host=dev-customerB sourcetype=typeA. Should be routed to index=customerB-keep
Event7 index=main host=dev-customerB sourcetype=typeB. Should be routed to index=customerB-keep

Any idea how this can be achieved with props.conf and transforms.conf (or by other means)?

gyslainlatsa
Motivator

hi mahesh_ravji,

This examples routes events of windows_snare_logsource type to the appropriate index based on their log types. "Application"logs will go to an alternate index, while all other log types, such as "Security", will go to the default index.
To make this determination, it uses props.conf to direct events of windows_snare_log source type through the transforms.conf stanza named "AppRedirect", where a regex then looks for the log type, "Application". Any event with a match on "Application"in the appropriate location is routed to the alternate index, "applogindex". All other events go to the default index.

  1. Edit props.conf
    Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

    [windows_snare_syslog]
    TRANSFORMS-index = AppRedirect

This directs events of windows_snare_syslog sourcetype to the AppRedirectstanza in transforms.conf

  1. Edit transforms.conf
    Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

    [AppRedirect]
    REGEX = MSWinEventLog\s+\d+\s+Application
    DEST_KEY = _MetaData:Index
    FORMAT = applogindex

This stanza processes the events directed here by props.conf. Events that match the regex (because they contain the string "Application" in the specified location) get routed to the alternate index, "applogindex". All other events route as usual to the default index.

try to use this example.

For more informations, follow this link:

http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Setupmultipleindexes

and read paragraph **Route specific events to a different index**

0 Karma

mahesh_ravji1
Explorer

Hi gyslainlatsa,

Please see my comment above...

0 Karma

somesoni2
SplunkTrust
SplunkTrust
0 Karma

mahesh_ravji1
Explorer

Thanks. I've looked at this but still cant figure out how to write the transform so that the REGEX looks at two fields before directing to a new index (i.e. the REGEX needs to look at the current index and host fields). Example if the index is NOT main, dont do a transform. If the index=main then set index to customerX portion from host field.

I need to apply the following transform but only if the current index=main, otherwise dont apply transform.

[override-index-by-host]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(?i)dev-(.+)$
FORMAT = $1
DEST_KEY = _MetaData:Index

0 Karma

gyslainlatsa
Motivator

find the regular expression that returns the events of the index main I call regex1.

next try like this:

1-props.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

 [Event_By_Index]
 TRANSFORMS-index = override-index-by-host

2-transform.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

[override-index-by-host]
SOURCE_KEY = MetaData:Host
REGEX = regex1                            // regular expression that returns the events
FORMAT = name_index               // name of the index that will receive data
DEST_KEY = _MetaData:Index
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...