Define sourctype based on the host - via a lookup?

[robf]

We have many different data sources which can only send on 514 UDP.

I need to define the sourcetype based on the host value.

I can see this can be done easily using a regex as described here

Advancedsourcetypeoverrides

However we have thousands of hosts with no obvious naming conventions.

so what i would want to do is maintain a list of hostnames and refernce that and say

if in ListA; sourcetype=typeA
if in ListB; sourcetype=typeB

etc

i really dont want to have several huge regex like

host1|host2|host3|host4|.................|host230|host231....

there is also no easy way to to regex based on the pattern of the events!

any ideas?!

[MuS]

Hi robf,

@Ayn and @kristian.kolb are both right, but you could try something like this and use a slim regex in transforms.conf:

[yourHostTransforms]
SOURCE_KEY = host
DEST_KEY = MetaData:Sourcetype
REGEX = host\d+
FORMAT = sourcetype::$1

I did something for an index re-write lately, but did not test it for sourcetype yet.

hope this helps ...

cheers, MuS

[MuS]

HeHe, let's call this selective memory 🙂 Only the regex example for the hosts was left while writing it. Never mind, maybe someone else can use this.

cheers, MuS

[kristian_kolb]

• "thousands of hosts with no obvious naming conventions".
• He wants 2 different sourcetypes.

But the post is almost a year old, and maybe he solved the problem already.

🙂

/K

[MuS]

Hi /k, I used the information available here in this post and showed an example what can be done. There is nothing mentioned about hosts not being named in such a fashion.

[kristian_kolb]

?? that would mean one sourcetype per host... assuming that you'd put the whole REGEX as capturing group.

And the hosts were not named in such a fashion. .. and @robf has already been down that road.

[kristian_kolb]

Ayn is right, unfortunately. You can set up Splunk to listen on more than one port (515,516,517 etc) and specify in inputs.conf on the receiving end that dictates the sourcetype to be used;

[udp://:514]
connection_host = dns
sourcetype = type_a

[udp://:515]
connection_host = dns
sourcetype = type_b

[udp://:516]
connection_host = dns
sourcetype = type_c

Unfortunately, you'd need to reconfigure a substantial number of your hosts to send to another port. And all data coming from each host would have the same sourcetype. (though you could override it, of course).

/K

[kristian_kolb]

Depending on your setup, i.e. if it's just the port that can't be changed, you might be able to do a similar operation, but by setting up a few syslog servers (rsyslog, splunk or other), and changing the destination ip on the sending hosts.

Then you install a forwarder on each syslog server... and set the sourcetype in inputs.conf. A bit messy - to say the least - for the initial setup, but hopefully fairly straightforward once it's set up.

/K

[robf]

thanks but unfortunately a number of devices cannot change their remote syslog port number

[Ayn]

Sorry, mechanisms like lookup are simply not available in that stage of the indexing pipeline. Lookups are pure search-time operations. Transforms like sourcetype rewriting, event filtering etc can be done with regexes only.

[robf]

thanks. so is the only way to do a huge regex as far as you know?