Getting Data In

Split Syslog (UDP:514) from multi hosts to multi indexes.

aaronnicoli
Path Finder

Hi there,

I am using syslog on Splunk currently to capture data from a piece of content-keeper hardware on our network that handles internet usage analysis. It's a fairly straight forward input, but, now I wish to capture more data from a firewall again using the standard UDP:514.

However, I would like for my CK data to go into splunk with the following values:

Host: content-keeper
Index: internet

And then my firewall to go in as:

Host: foo
Index: firewalls

Currently my inputs.conf segment looks like:

[udp://514]
connection_host = none
host = content-keeper
index = internet
sourcetype = content-keeper
source = content-keeper

props.conf

[content-keeper]
LOOKUP-domaingroups = common_domains basedomain AS basedomain OUTPUTNEW domaingroup AS domaingroup

transforms.conf

[common_domains]
filename = group_common_domains.csv

As you can see with props and transforms, I am just doing some basic lookups.

Any help would be very very appreciated.

Aaron.

1 Solution

jbsplunk
Splunk Employee
Splunk Employee

What you're looking for is documented here:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

What you'd need to do would be something like this:

props.conf:

[content-keeper]
TRANSFORMS-index = firewalls,hosts

transforms.conf:

[firewalls]
REGEX = <MYFIREWALLREGEX>
DEST_KEY = _MetaData:Index
FORMAT = firewalls

[hosts]
REGEX = <(MYFWHOSTREGEX)>
DEST_KEY = MetaData:Host
FORMAT = host::$1

By the way, at the time you do this, the 'firewalls' index must have already been created.

View solution in original post

bstimely
New Member

I would suggest setting up a syslog server that captures all the incoming syslog messages and stores them into a unique directory or file for each host. This is pretty straightforward of a syslog setup. I use a directory with the hostname and the hostname-date as the log file for each day.
Then create file monitors for each of the directories. You can make all for determination about which index to put things in using your inputs.conf.

0 Karma

Runals
Motivator

Create multiple DNS entries, assign them to virtual hosts on your system, configure syslog to look for UDP/514 from each of those IPs and write the data to disk, then configure multiple Splunk imports to look at each of the paths syslog writes to and assign them to the index of choice. Probably sounds complicated but isn't bad really depending on what level of access or turn around you have to things like new DNS entries, OS, etc.

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

What you're looking for is documented here:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

What you'd need to do would be something like this:

props.conf:

[content-keeper]
TRANSFORMS-index = firewalls,hosts

transforms.conf:

[firewalls]
REGEX = <MYFIREWALLREGEX>
DEST_KEY = _MetaData:Index
FORMAT = firewalls

[hosts]
REGEX = <(MYFWHOSTREGEX)>
DEST_KEY = MetaData:Host
FORMAT = host::$1

By the way, at the time you do this, the 'firewalls' index must have already been created.

aaronnicoli
Path Finder

Solved with a simple regex that can determine which data to perform the change on and which to not perform changes on.

0 Karma

aaronnicoli
Path Finder

Thanks JB,

This is pretty much what I have come up with. The issue I am encountering now is with the regex, I cannot seem to find a way of extracting the hostname or some indicator of the orgin host. When I removed the "host=" over-ride from my inputs.conf, the host value simply defaulted to the fqdn of the splunk indexer...
I am going to simply have to look more into it.

Thanks very very much for you well written response!
Aaron.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...