Getting Data In

How to send syslog data to a specific index based on a string in the log?

hlarimer
Communicator

I have syslog data coming to a distributed environment. I am trying to send the data to a specific index based on a string that is in the data. The string is "NetScreen" and the index is juniper. I have the following props.conf and transforms.conf deployed to the forwarder that is collecting these logs but the logs are still coming into index main

Props.conf

[JuniperFW]
Transforms-JuniperFW=JuniperFW

Transforms.conf

[JuniperFW]
DEST_KEY = _MetaData:Index
REGEX = (NetScreen)
FORMAT = index::juniper
0 Karma
1 Solution

acharlieh
Influencer

"The forwarder that is collecting these logs" -> Is this a Universal Forwarder or a Heavy Forwarder? The UF only does input phase. In which case you would have to set this on your Indexer(s) or Heavy Forwarders. See the Splunk Wiki for a handy reference for what applies where.

Also you probably want to capitalize TRANSFORMS in the props.conf

View solution in original post

acharlieh
Influencer

"The forwarder that is collecting these logs" -> Is this a Universal Forwarder or a Heavy Forwarder? The UF only does input phase. In which case you would have to set this on your Indexer(s) or Heavy Forwarders. See the Splunk Wiki for a handy reference for what applies where.

Also you probably want to capitalize TRANSFORMS in the props.conf

hlarimer
Communicator

Do you know if I can also change the sourcetype here. I have tried using another stanza in transforms.conf and props.conf as shown below, but it is not changing the sourcetype, is there another method I should be using?

props.conf
[JuniperFW]
TRANSFORMS-JuniperFW=JuniperFW

[JuniperFW_ST]
TRANSFORMS-JuniperRW_ST=JuniperFW_ST

transforms.conf
[JuniperFW]
DEST_KEY = _MetaData:Index
REGEX = (NetScreen)
FORMAT = juniper

[JuniperFW_ST]
DEST_KEY = MetaData:Sourcetype
REGEX = (NetScreen)
FORMAT = JuniperFW

0 Karma

acharlieh
Influencer
0 Karma

acharlieh
Influencer

You should be able to change sourcetype at index time as well... (I'm not sure if the other sourcetype's props would apply or not when you change it, I haven't done much in this space). But I will note that according to the transforms.conf doc setting MetaData:Sourcetype requires the value to be prefixed sourcetype:: so your FORMAT line should be sourcetype::JuniperFW

Alternatively, you can also rename sourcetypes at search time (slightly different configs).

0 Karma

hlarimer
Communicator

I ended up leaving them in index=main and then setting the sourcetype and index time. My ultimate goal was to normalize the data to follow the CIM so I could create searches across multiple types of firewalls. With the data separated into sourcetypes I am now able to create the search time field extractions I needed, although I struggled with this until I realized that the app that has the field extractions needed to be set to be shared globally in metadata/default.meta.

Everything is working now and now its time to start actually using the data!!

0 Karma

hlarimer
Communicator

It is a UF...
Moving those to the Indexer took care of it! I had to change FORMAT = index:juniper to FORMAT = juniper but that got the data going to right location. Thank you for your help!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...