Getting Data In

Change sourcetype via field extraction and transforms

niboucher
Explorer

Hi there,

One of UF is configured to send logs to sourcetype testData.
I'd like to push some of those logs matching a certain pattern (all logs matching the "[A][B]" pattern) to sourcetype testData_B.

Sample of log

[A][B] blabla
[A][C] blabla

I tried to use transforms and field extraction but I couldn't make it work. I don't have ssh access so I did via the web interface

Transformation
alt text

Field extraction
alt text

What's wrong with my setup?

Thanks!

Tags (1)
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I have ingested below data in my lab environment with below config and it is working fine (Splunk version 7.1.2)

[A][B] blabla
[A][C] blabla

props.conf (In below config I have given DATETIME_CONFIG = CURRENT because in sample data we don't have any timestamp)

[testData]
TRANSFORMS-testProp = testData_trans
SHOULD_LINEMERGE = False
DATETIME_CONFIG = CURRENT
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

transforms.conf

[testData_trans]
DEST_KEY = MetaData:Sourcetype
REGEX = \[A\]\[B\]
FORMAT = sourcetype::testData_B

With above config [A][B] blabla line indexed with sourcetype testData_B and [A][C] blabla line indexed with sourcetype testData

0 Karma

niboucher
Explorer

I tried to modify directly props.conf and transforms.conf and restart splunk but that didn't make it

transforms.conf

[testData_trans]
SOURCE_KEY = MetaData:Sourcetype
REGEX = (\[A\]\[B\].*)
FORMAT = sourcetype::testData_B

props.conf

 [sourcetype::testData]
 TRANSFORMS-testProp = testData_trans

Do you guys have any idea that could help me make this work?

0 Karma

niboucher
Explorer

Thanks a lot man!
The problem was certainly caused by the lack of timestamp.
Setting it to CURRENT and using the regex you provided made it work.

Niboucher

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Great that it worked, DATETIME_CONFIG = CURRENT assigns timestamp from splunk server itself, which means when data indexed whatever time present on splunk server that time will be assigned to events.

I have converted my comment to answers, so that you can accept & upvote it.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Config should be like this

props.conf

[testData]
TRANSFORMS-testProp = testData_trans

transforms.conf

[testData_trans]
DEST_KEY = MetaData:Sourcetype
REGEX = (\[A\]\[B\].*)
FORMAT = sourcetype::testData_B
0 Karma

niboucher
Explorer

That didn't work either.

UF conf
inputs.conf

[monitor:///var/log/containers/*.log]
sourcetype = testData
recursive = false

Indexer conf
props.conf

 [testData]
 TRANSFORMS-testProp = testData_trans

transforms.conf

 [testData_trans]
 DEST_KEY = MetaData:Sourcetype
 REGEX = (\[A\]\[B\].*)
 FORMAT = sourcetype::testData_B
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

It should work, can you please try with REGEX = \[A\]\[B\] in transforms.conf, don't forget to restart splunk.

Additionally data which is already indexed with old sourcetype will not change only new data which will come to Indexer now onwards will have new sourcetype (Here I am guessing that UF is sending logs to Indexer directly and not via Heavy Forwarder)

0 Karma

niboucher
Explorer

it didn't work

When I use regex below, it works and index everything in the dest sourcetype but adding a pattern inside the capturing group breaks it
REGEX= (.*)

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @niboucher,

Do you want to index data in different sourcetype when it comes to Indexer ? Are you running single instance, distributed environment or clustered environment ?

If you don't have ssh access to splunk server you can use REST API to do configuration on indexers but this approach is not suitable for clustered indexer, you can find Splunk SDK Python script on this answer to create props.conf configuration, similarly you can create configuration for transforms.conf (Here is another answer which will help you to do configuration which you will require to rename sourcetype based on REGEX)

0 Karma

niboucher
Explorer

Hey,

Exactly!
I want data to be indexed in a different sourcetype when data arrives to the indexer.
It's a clustered environnment and I have very limited rights as a user.
REST API doesn't seem to be a suitable option.

I will deploy a new splunk instance with full admin rights to try to modify directly transforms.conf and props.conf

Thanks

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...