Getting Data In

How to forward logs with Splunk Universal Forwarder for the files with no header and logs should be in form of key/value

anupam491
Engager

I have a Splunk Forwarder setup already on my host.

I have certain files on folder (/tom/mike/). File names are starting with Back*.

The content of file may in one or multiple line. There are multiple fixed position values in each line with no header.

Content (Example: Consider "-" as one space)

Tom---516-----RTYUI------45678
Mik---345-----XYXFF------56789

I need splunk logs for each line.

Like:

Key1= Tom   Key2=516   Key3= RTYUI  Key4= 45678

Key1= Mike  Key2= 345  Key3= XYXFF  Key4= 56789

I know inputs.conf changes would be like below.

[monitor:///tom/mike/Back*]
index=myIndex
blacklist=\.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData

Please suggest changes which can be done in props.conf. Please keep in mind that delimiter is fixed for each value in line but its not same (like 2 spaces) for all column values. There are no headers as well in these files.

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

Here's how I would do it:

On the search heads, create an app with a props.conf:

 [BackFileData]
 EXTRACT-fields=^(?<Key1>\S+)\s+(?<Key2>\d+)\s+(?<Key3>\S+)\s+(?<Key4>\d+)

Then I'd make the permissions on the app "global" if I wanted the extraction to work for anyone in any app, "app level" if I only wanted it to work inside the app, or "private" if I only wanted myself to have the extraction (in any app).

Doing it this way will apply the schema at search time which is the best practice versus transforming the data and indexing the fields.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Here's how I would do it:

On the search heads, create an app with a props.conf:

 [BackFileData]
 EXTRACT-fields=^(?<Key1>\S+)\s+(?<Key2>\d+)\s+(?<Key3>\S+)\s+(?<Key4>\d+)

Then I'd make the permissions on the app "global" if I wanted the extraction to work for anyone in any app, "app level" if I only wanted it to work inside the app, or "private" if I only wanted myself to have the extraction (in any app).

Doing it this way will apply the schema at search time which is the best practice versus transforming the data and indexing the fields.

anupam491
Engager

How to make the permissions on the app "global"? I tried without any permission setup and got the same result.

0 Karma

anupam491
Engager

I am trying to go by first suggestion.
Directory

~/splunk/heavyforwarder/etc/apps/Splunk_TA_nix/default

props.conf

[BackFileData]
TRANSFORMS-backfiledata=BackFileData_main_fields

transforms.conf

[BackFileData_main_fields]
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4

inputs.conf

[monitor:///tom/mike/Back*]
index=cs_it
blacklist=.(gz|zip|bkz|arch|etc)$
sourcetype = BackFileData

I am still getting logs like below.

8/9/17
3:58:38.000 PM Tom 516 RTYUI 45678

Can someone please tell that what is wrong in transforms.conf/props.conf ?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Instead of the \s{x} where x is a number, try \s+. Same for \w and \d.

0 Karma

jkat54
SplunkTrust
SplunkTrust

BTW, your logs will remain the same... it's the fields Key1-Key4 that you're creating here.

Are you trying to transform your data instead?

If so you will need SEDCMD in props.conf instead:

SEDCMD-keys=s/(\S+)\s+(\d+)\s+(\S+)\s+(\d+)/Key1=\1 Key2=\2 Key3=\3 Key4=\4/g

You won't need the transforms.

You will have to index new data to see the props applied, and the props need to be on the forwarder.

0 Karma

anupam491
Engager

Thanks. Now keys are coming.

It is coming like

8/9/17
5:11:34.000 PM

Key1=Tom Key2=516 Key3=RTYUI Key4=45678
Key1=Mik Key2=516 Key3=RTYUI Key4=45678

But I want to forward each line separately like:

8/9/17
5:11:34.000 PM

Key1=Tom Key2=516 Key3=RTYUI Key4=45678

8/9/17
5:11:34.000 PM

Key1=Mik Key2=516 Key3=RTYUI Key4=45678

0 Karma

jkat54
SplunkTrust
SplunkTrust

Add this to the props

 SHOULD_LINEMERGE=false
  LINE_BREAKER=([\n\r]+)
0 Karma

jkat54
SplunkTrust
SplunkTrust

You go to the little cog / gear next to apps on the home screen (app manager) and you click permissions next to the app. Don't forget to restart Splunk too.

0 Karma

anupam491
Engager

I have checked and found this is as "app only" and I think that is correct.

Sorry to bother you but these steps didn't work. Can you please explain steps considering my initial questions.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If you change it to all apps that makes it global. Otherwise you will have to be in the app you created when you search for the extractions to work.

0 Karma

woodcock
Esteemed Legend

As long as the spacing is consistent, just use traditional RegEx means like this in transforms.conf:

[BackFileData_main_fields]
#Tom---516-----RTYUI------45678
REGEX = ^(\w{3})\s{3}(\d{3})\s{5}(\w{5})\s{6}(\d+)$
FORMAT = Key1::$1 Key2::$2 Key3::$3 Key4::$4
0 Karma

anupam491
Engager

I assume that _main_fields is a keyword here.

So BackFileData_main_fields means sourcetype_main_fields ?

0 Karma

anupam491
Engager

Still logs are coming as a whole without keys.

I have done these changes in props.conf and transforms.conf. Regex is correct as I have checked that.

0 Karma

anupam491
Engager

I think some more changes are required. Right now both lines are coming together without keys.

0 Karma

woodcock
Esteemed Legend

You need to deploy to all indexers, restart all Splunk instances there, and then test by ONLY looking at post-restart events (old events will stay broken).

0 Karma

jkat54
SplunkTrust
SplunkTrust

You'd need this in props.conf to enable the answer by @woodcock

[BackFileData]
TRANSFORMS-backfiledata=BackFileData_main_fields

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...