What is the best practice for assigning indexes and sourcetypes for multiple UDP feeds to the same port?

[juanlazarosanch]

Excuse the ignorance, I'm coming from the syslog-ng world and just now starting with Splunk.

I have multiple devices in my environment that are sending their logs to our Splunk indexer via UDP 514. So if I have the following devices:

router1
asa1
switch1

and I want to assign each the following index and sourcetypes:

router1 - index=network, sourcetype=cisco:ios
asa1 - index=network_firewall, sourcetype=cisco:asa
switch1 - index=network, sourcetype=cisco:ios

What is the best way to go about doing this? I've been reading stuff here and there (some of it conflicting) such as I'd need to edit/add several entries in inputs.conf, transforms.conf, and props.conf from /opt/splunk/etc/system/local...OR, if I'm just using the Search App, I can do something like this in the /opt/splunk/etc/apps/search/local/inputs.conf file:

[udp://172.16.1.1:514]
sourcetype = cisco:asa
index = network_firewall
host = asa1.example.com

[udp://172.16.1.27:514]
sourcetype = syslog
index = network
host = switch1.example.com

I'm hoping you can steer me in the right direction and guide me toward best practices. Thanks.

[hgrow]

This guy wrote the best artical I've seen so far:

http://www.georgestarcher.com/splunk-success-with-syslog/

Greetings

[sjohnson_splunk]

The best practice would be to use a separate syslog or rsyslog server to collect the UPD data and write it to disk. Then use a UF to monitor the files.

If that is not possible, then the above technique is what you need to do. There are a couple problems with the transforms.conf settings.

The DEST_KEY value for over-riding sourcetype does not begin with an _
Also you will need to add a sourcetype: to the string in the FORMAT statement:

FORMAT = sourcetype::cisco:asa

From the transforms.conf.spec file:

MetaData:Sourcetype : The sourcetype of the event.
The value must be prefixed by "sourcetype::"

[juanlazarosanch]

Of the following two directories, which settings take precedence? Should I always try to modify the local directory vs the specific app directory?
/opt/splunk/etc/system/local
/opt/splunk/etc/apps/search/local/ (would I even need to worry about the conf files in this directory?)

I'm assuming I would need to do the same for the other network equipment (switch, router) which I wanted to have an index of network and a sourcetype of syslog. Is that correct? Thanks for your answers, friends!

[sjohnson_splunk]

etc/system/local over-rides everything

I would do this inside an app, search is ok or create your own barebones app and put the settings in a local directory.

Some network gear can send to ports other than 514 so you might want to send all devices of one type to a unique port (1514, 2514, etc.) and make it easier on yourself.

[juanlazarosanch]

Currently, this stanza exists inside /opt/splunk/etc/apps/search/local/inputs.conf (I inherited this Splunk environment). Would I need to remove it so that the /opt/splunk/etc/apps/search/local/transforms.conf and /opt/splunk/etc/apps/search/local/props.conf can properly set the index, sourcetype or can I leave it as is? In other words, will transforms.conf and props.conf override the inputs.conf file?

[udp://514]
connection_host = dns
sourcetype = access_combined
index = network

[twinspop]

Including the remote address in the UDP input stanza is deprecated. Instead, you'll want to use transforms to change the index and/or sourcetype.

In transforms.conf, something like this:

[sendto_network_firewall]
SOURCE_KEY = host
REGEX = ^172\.16\.1\.1$
DEST_KEY = MetaData:Index
FORMAT = index::network_firewall


[change_asa_sourcetype]
SOURCE_KEY = host
REGEX = ^172\.16\.1\.1$
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::cisco:asa

In props.conf, you'd reference these stanza names in a TRANSFORM entry:

[source::udp:514]
TRANSFORM-asa_fixups = sendto_network_firewall, change_asa_sourcetype
TRANSFORM-other_fixup = sendto_other_index, change_another_sourcetype

It should be noted that changing sourcetypes is kinda messy. Many pieces of config reference sourcetype. Do they fire before or after your change? It makes things more complicated, prone to mistakes, and harder to troubleshoot. Just my 2 cents.

EDIT: fixed per sjohnson's post. Good catch!

[juanlazarosanch]

Your comment got me a little worried. What's your recommended solution? The same one sjohnson suggested? Thanks for your advice!

[twinspop]

Yes, his suggestions are good: Either setting up a syslog server, or sending to different ports for different devices/sourcetypes.