Splunk Search

Help parsing out events - need to get timestamp, host and sourcetype to rewrite metadata

brent_weaver
Builder

I have the following event:

{   [-] 
     ident:  vcap.cloud_controller_ng   
     message:     {"timestamp":1489461920.4637804,"message":"(0.000343s) SELECT * FROM `spaces` WHERE `id` = 479","log_level":"info","source":"cc.db","data":{"request_guid":"27de1815-57c0-41a5-63e1-614c44dfcac7::ef9ddb89-8aaf-46da-8843-902a91f95b44"},"thread_id":47430748867200,"fiber_id":47430746457820,"process_id":5696,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.29.0/lib/sequel/database/logging.rb","lineno":70,"method":"block in log_each"}  
     orig_host:  10.72.134.207  
     pid:    job=api_z2 index=1 
     pri:    14 
}

I need to use the ident field as the source type, get the timestamp out of the message line and set host as org_host field. I would also like to parse out the message field and make it kv pairs in son format. I have tried numerous things to no avail. How would you all approach this?

Any help is much appreciated!

0 Karma

brent_weaver
Builder

I solved this by the following:

props:

[test]
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = json
TRANSFORMS-updateMetaData = autosource, autohost, autoparse, fixsource

transforms:

[autosource]
SOURCE_KEY = field:ident
REGEX = (.*)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[autohost]
SOURCE_KEY = field:orig_host
REGEX = (.*)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[autoparse]
SOURCE_KEY = field:message
REGEX = (.*)
FORMAT = $1
DEST_KEY = _raw

[fixsource]
SOURCE_KEY = field:source
REGEX = (.*)
FORMAT = source::$1
DEST_KEY = MetaData:Source

I will admit the fixsource does not work but the rest does. I was missing the INDEXED_EXTRACTIONS = json

0 Karma

woodcock
Esteemed Legend

Give this a try (there are SEVERAL differences from the other answer):

input.conf (on forwarder monitoring the file)

[monitor...
.....
sourcetype=someGenericName

props.conf (on heavy forwarder/Indexer)

[someGenericName]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?ms)(((?:^|[\r\n\s]+}[\r\n\s]+){[^\r\n]+[\r\n\s]+)|[\r\n\s]+})
TIME_PREFIX = {\"timestamp\"\:
TIME_FORMAT = %s.%N
MAX_TIMESTAMP_LOOKAHEAD = 18
TRANSFORMS-changest = set_sourcetype_my_log

transforms.conf (on heavy forwarder/Indexer)

[set_sourcetype_my_log]
REGEX = ident\s*:\s*(\S+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

props.conf (on Search Head)

[host::YourHostHere]
REPORT-extractcv = first_level_kvps, second_level_kvps, third_level_kvps

transforms.conf (on Search Head)

[first_level_kvps]
REGEX = (?ms)\s+([^\r\n:]+)\s*:\s*{?([^\r\n]+)(?<![}\s])
FORMAT = $1::$2

[second_level_kvps]
SOURCE_KEY = message
REGEX = (?ms)"([^"]+)":([^,]+)
FORMAT = $1::$2

[third_level_kvps]
SOURCE_KEY = data
REGEX = (?ms)"([^"]+)":([^,]+)
FORMAT = $1::$2
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Give this a try

input.conf (on forwarder monitoring the file)

[monitor...
.....
sourcetype=someGenericName

props.conf (on heavy forwarder/Indexer)

[someGenericName]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\{\s*[\r\n]*ident:)
TIME_PREFIX = timestamp\"\:
TIME_FORMAT = %s.%N
MAX_TIMESTAMP_LOOKAHEAD = 18
TRANSFORMS-changest = set_sourcetype_my_log

transforms.conf (on heavy forwarder/Indexer)

[set_sourcetype_my_log]
REGEX = ident\:\s*(\S+)
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

props.conf (on Search Head)

[someGenericName]
REPORT-extractcv = colon_delim_fields

transforms.conf (on Search Head)

[colon_delim_fields]
DELIMS = ",", ":"
0 Karma

brent_weaver
Builder

Hey thanks for the info but it did not work. I am using a standalone instance in a VM on my laptop. But I did combine transforms and props since this is an indexer and sh.

Any other thoughts?

0 Karma

brent_weaver
Builder

*parsing not passing

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...