Getting Data In

Updating metadata sourcetype with data from events

brent_weaver
Builder

Hello all... I have the following file:

conn.log: 1486576311.492453 Cid7Nq2yj6VZ3FdO8b  10.28.7.27  39525   10.12.7.17  8080    tcp -   -   -   -   OTH T   T   0   C   0   0   0   0   (empty)

I need to carve the first element (conn.log) delimited by . to be my source type. This value can vary but it will always have a . as a delimiter. Here is my props.conf:

[bro]
REPORT-format = BroAutoSrc, TrashComments
SHOULD_LINEMERGE = false
TRUNCATE = 0
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %s.%6N

And transforms.conf:

splunk[/opt/splunk/etc/apps/bro/default] # cat transforms.conf 
[BroAutoSrc]
DELIMS = "."
FIELDS = "orig_source"
REGEX = (\.[a-zA-Z0-9]+\.)?([a-zA-Z0-9]+)
FORMAT = sourcetype::bro_$1
DEST_KEY = MetaData:Sourcetype
WRITE_META = true

[TrashComments]
REGEX = ^\s*#
DEST_KEY = queue
FORMAT = nullQueue

This is not working. What am I doing wrong? Does the source typing have to take place on indexing or can it be done at search time? Any help is much appreciated, thanks.

0 Karma

starcher
SplunkTrust
SplunkTrust

Sourcetype like time, line breaking, source etc are index time fields not search time. Bro is a high volume data source. I would not recommend index time rewriting of sourcetype. Just set the sourcetype correctly at inputs.

0 Karma

brent_weaver
Builder

That is GREAT feedback and I did wonder about that! I was trying to basically recreate the Bro for IDS Splunk app since the sensors can generate allot of different kinds of files with different fields. The issue we have with this model is that we have the bro logs aggregating into our syslog-ng server so we lose the headings that are generated in the files, which is what the app uses to name fields/keys. I cannot imagine I am the first splunker to encounter this issue.

Any thoughts are more than welcome!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...