Splunk Search

How can I override source type?

ninisimonishvil
Path Finder

Hello

I have an event that starts like this:

02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:07:32 10.5.0.11 AlteonOS :

10.5.0.1 58696 10.5.0.101 80 tcp 12/02/2018-17:07:10 12/02/2018-17:07:11 10.6.0.101 80 0.0.0.0 2060 |

10.5.0.1 58697 10.5.0.101 80 tcp 12/02/2018-17:07:10 12/02/2018-17:07:11 10.6.0.101 80 0.0.0.0 2075 |

I want to override the sourcetype while indexing. Here are my transforms and props file contents:

Transforms:
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

Props
[syslog3.txt]
TRANSFORMS-sourcetype = sourcetypechange

syslog3.txt is a source, I upload it to test if there are changes in sourcetype, no success. Can anyone tell me if I'm doing anything wrong?

0 Karma

ninisimonishvil
Path Finder

PROPS
[source::C:\Users\Administrator\Desktop\test.txt]
TRANSFORMS-sourcetype = sourcetypechange

TRANSFORMS

[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

I'm changing sourcetype because need to to some field and timestamp extractions however not for all events, but only for those that start with : 02-02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:09:32 10.5.0.11 AlteonOS

I'm using test.txt file to test if the sourcetype override works (I keep uploading it after every change, and keep restarting splunk). No result so far.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ninisimonishvili,
if you put [syslog3.txt] in a props.conf stanza, Splunk takes it as a sourcetype.
You should use

[source::<source>]

using the full path of your source

See https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf

Bye.
Giuseppe

0 Karma

ninisimonishvil
Path Finder

Thank you Giuseppe,

I tried indicating full path (also used sourcetype instead of source) however no success.
The regex that I'm using in Transforms - shall it describe the whole event or just the beginning?

because the start of the event will follow this sequence
: 02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:07:32 10.5.0.11 AlteonOS :

however, the number of followed IP addresses and timestamps may vary from event to event.

0 Karma

serjandrosov
Path Finder

Hi ninisimonishvili,
How your full path stanza looks now?

Also could you please explain why you choose this solution? Do you have different sourcetypes in one dataflow? Or maybe you can't change it at monitor stanza level?

If you need only to rename existing sourcetype, you could do that by using splunk sourcetype rename feature.
https://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Renamesourcetypes

Please keep in ming that REGEX can put some overhead to indexer depending on EPS.

0 Karma

ninisimonishvil
Path Finder

Hello,

to clarify, I need to change sourcetype in order to make some field and correct timestamp extraction, however I need to make these extraction not for all the event from a particular sourcetpe, that is why I'm describing sequence of event via regex where I need to change sourcetype.

now I'm just testing the syntax via test.txt file. keep uploading it after conf file alterations and restart. however getting no results.

here are the conf file configurations:

PROPS
[source::C:\Users\Administrator\Desktop\test.txt]
TRANSFORMS-sourcetype = sourcetypechange

TRANSFORMS
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog

I checked regex and it is fine.

0 Karma

serjandrosov
Path Finder

Regarding
https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf

Considerations for Windows file paths:

When you specify Windows-based file paths as part of a [source::] stanza, you must escape any backslashes contained within the specified file path.

Example: [source::c:\path_to\file.txt]

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ninisimonishvili,
the regex to find could be in every part of your event: you can verify it in Splunk using the rex command:

index=my_index | rex "my_regex"

in this way you should have as results the event to discard.

The job is to find (if exist) the correct regex to find all the events to discard: you could also think to use more than one regex, anyway the method is the one above: use rex command in Splunk search, eventually more than one.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...