Getting Data In

Rule based source typing

FaceF18
New Member

I'm trying to set the sourcetype on some events I get based on their contents, and then I want to send each of those differentiated sourcetypes to their own indexes. I've tried a bunch of different ways, and none of my approaches seem to work quite like the docs say they should.

So, for starters, source typeing. I feel like what I'm trying to do is simple. If the string FlightEvent occurs anywhere in the event, it should be a FlightEvent. Flight and Event are actually separate xml opening tags, but I can't seem to get less-than and greater-than symbols to display in markdown. I don't know if that has any impact in props or transforms.conf.

In props.conf

[FlightEvent]
TRANSFORMS-flighteventtrans = flighteventformat

In transforms.conf

[flighteventformat]

REGEX = FlightEvent

LOOKAHEAD = 16

DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype:FlightEvent

No good.

I tried setting up rule based source typing.

In props.conf

[rule::flighteventrule]

sourcetype=FlightEvent

MORE_THAN_1 = FlightEvent

No good. I also can't get sourcetypes to go to the correct indexes, or actually any index other than main, but I guess I'll try to deal with that when I get source typing figured out.

Tags (1)
0 Karma

lguinn2
Legend

The symbols < and > are special characters in regular expressions. Also, your props.conf has a problem - you are asking it to process events of sourcetype FlightEvent - before you have assigned the sourcetype of FlightEvent!!

I suggest this in your props.conf

[source::yourinputsourcefilehere]
TRANSFORMS-flighteventtrans = flighteventformat

In transforms.conf

[flighteventformat]
REGEX = \<Flight\>\<Event\>
LOOKAHEAD = 16
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype:FlightEvent

Skip the rule-based sourcetyping, you don't need it. Also the REGEX above may be wrong, because I don't entirely understand the format of your events. If you could post a sample of the raw data (anonymized of course), that would be really helpful. If you paste it into the question, you can highlight it and then click on the 101 icon to keep the markdown from messing with it.

0 Karma

FaceF18
New Member

I get flight events from a number of different sources and hosts, so I'm not sure how to write the first statement so that it applies in this case. I tried both source and host * and .* with the hope that this would apply that transform to every event, and then I edited the regex to escape any special characters, and I still didn't have any luck.

Also, I can't edit my post right now because every captcha is coming back invalid, but basically I want any event that has this substring "<Flight><Event>" in it anywhere to be set as a FlightEvent. Thanks for the help.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...