Hello
I have an event that starts like this:
02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:07:32 10.5.0.11 AlteonOS :
10.5.0.1 58696 10.5.0.101 80 tcp 12/02/2018-17:07:10 12/02/2018-17:07:11 10.6.0.101 80 0.0.0.0 2060 |
10.5.0.1 58697 10.5.0.101 80 tcp 12/02/2018-17:07:10 12/02/2018-17:07:11 10.6.0.101 80 0.0.0.0 2075 |
I want to override the sourcetype while indexing. Here are my transforms and props file contents:
Transforms:
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog
Props
[syslog3.txt]
TRANSFORMS-sourcetype = sourcetypechange
syslog3.txt is a source, I upload it to test if there are changes in sourcetype, no success. Can anyone tell me if I'm doing anything wrong?
PROPS
[source::C:\Users\Administrator\Desktop\test.txt]
TRANSFORMS-sourcetype = sourcetypechange
TRANSFORMS
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog
I'm changing sourcetype because need to to some field and timestamp extractions however not for all events, but only for those that start with : 02-02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:09:32 10.5.0.11 AlteonOS
I'm using test.txt file to test if the sourcetype override works (I keep uploading it after every change, and keep restarting splunk). No result so far.
Hi ninisimonishvili,
if you put [syslog3.txt]
in a props.conf stanza, Splunk takes it as a sourcetype.
You should use
[source::<source>]
using the full path of your source
See https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf
Bye.
Giuseppe
Thank you Giuseppe,
I tried indicating full path (also used sourcetype instead of source) however no success.
The regex that I'm using in Transforms - shall it describe the whole event or just the beginning?
because the start of the event will follow this sequence
: 02-12-2018 17:07:33 Local7.Info 10.5.0.11 Feb 12 17:07:32 10.5.0.11 AlteonOS :
however, the number of followed IP addresses and timestamps may vary from event to event.
Hi ninisimonishvili,
How your full path stanza looks now?
Also could you please explain why you choose this solution? Do you have different sourcetypes in one dataflow? Or maybe you can't change it at monitor stanza level?
If you need only to rename existing sourcetype, you could do that by using splunk sourcetype rename feature.
https://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Renamesourcetypes
Please keep in ming that REGEX can put some overhead to indexer depending on EPS.
Hello,
to clarify, I need to change sourcetype in order to make some field and correct timestamp extraction, however I need to make these extraction not for all the event from a particular sourcetpe, that is why I'm describing sequence of event via regex where I need to change sourcetype.
now I'm just testing the syntax via test.txt file. keep uploading it after conf file alterations and restart. however getting no results.
here are the conf file configurations:
PROPS
[source::C:\Users\Administrator\Desktop\test.txt]
TRANSFORMS-sourcetype = sourcetypechange
TRANSFORMS
[sourcetypechange]
REGEX = \d{2}-\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2}\s\Local7.Info\s\d*.\d*.\d*.\d*\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\d*.\d*.\d*.\d*\s\w\lteonOS\s\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Redsyslog
I checked regex and it is fine.
Regarding
https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Propsconf
Considerations for Windows file paths:
When you specify Windows-based file paths as part of a [source::] stanza, you must escape any backslashes contained within the specified file path.
Example: [source::c:\path_to\file.txt]
Hi ninisimonishvili,
the regex to find could be in every part of your event: you can verify it in Splunk using the rex command:
index=my_index | rex "my_regex"
in this way you should have as results the event to discard.
The job is to find (if exist) the correct regex to find all the events to discard: you could also think to use more than one regex, anyway the method is the one above: use rex command in Splunk search, eventually more than one.
Bye.
Giuseppe