Getting Data In

How to edit my transforms.conf in order to set the sourcetype for each syslog being forwarded?

a548506
Path Finder

Hello,

I have a customer sending three different kind of logs via syslog. I am pulling the logs off of a network feed where I had him point the syslogs to. It's listening on port xx514. The logs look like this:

SplunkSystem: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time
SplunkActivity: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time
SplunkAudit: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time

I have the syslogs going to a temp sourcetype then I am grabbing that and using a transforms to set the sourcetype for each syslog.

My inputs:

[udp://xx514]   
index=ABC
sourcetype=abc_temp_syslog
connection_host = dns
disabled = 0

My Props:

[abc_temp_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
TRANSFORMS-set1= changesourcetypetosystem
TRANSFORMS-set2= changesourcetypetoaudit
TRANSFORMS-set3= changesourcetypetoactivity

[abc_system_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false

[abc_audit_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false

[abc_activity_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false

My Transforms.conf

[changesourcetypetoactivity]
REGEX = SplunkActivity
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::abc_activity_syslog

[changesourcetypetosystem]
REGEX = SplunkSystem
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::abc_system_syslog

[changesourcetypetoaudit]
REGEX = SplunkAudit
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::abc_audit_syslog

So what Splunk is doing is putting all three syslogs into the temp_syslog and is ignoring the transforms ... which leads me to believe that it is my transforms.conf that has some kind of error.

Thanks for the help!!

1 Solution

sloshburch
Splunk Employee
Splunk Employee

I'd try debugging to see which transform is not working by removing all and adding back one at a time. Easier to debug one than all three. Also, I think you can collapse them:

 TRANSFORMS-setST= changesourcetypetosystem, changesourcetypetoaudit, changesourcetypetoactivity

For reference, we're talking about this topic: http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

I'd also try regex like:

REGEX = ^SplunkActivity\:.*

That should look for it at the start of the event only.

View solution in original post

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I'd try debugging to see which transform is not working by removing all and adding back one at a time. Easier to debug one than all three. Also, I think you can collapse them:

 TRANSFORMS-setST= changesourcetypetosystem, changesourcetypetoaudit, changesourcetypetoactivity

For reference, we're talking about this topic: http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

I'd also try regex like:

REGEX = ^SplunkActivity\:.*

That should look for it at the start of the event only.

0 Karma

a548506
Path Finder

Thanks to you both! It's working as intended.

Just updated the props to the following:

[abc_temp_syslog]
KV_MODE = auto
ANNOTATE_PUNCT = false
TIME_PREFIX = ^
TIME_FORMAT = %m $d %H:%M:%S
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
TRANSFORMS-set1= changesourcetypetosystem, changesourcetypetoaudit, changesourcetypetoactivity

...and used this REGEX in the transforms for the corresponding messages:
REGEX = SplunkSystem
REGEX = SplunkActivity
REGEX = SplunkAudit

Thanks Again Burch and lguinn.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I would recommend reviewing which of the props you are using default values for AND perhaps remove those and let it infer from the default. It makes it easier to maintain later so you know you didn't actually care that KV_MODE = auto (for example) and that you were just using the defaults. (Check the props.conf.spec file to see that many of them are using the defaults)

Did you really want ANNOTATE_PUNCT turned off? With what I know of your environment, I'm surprised to see such an advanced setting used.

0 Karma

lguinn2
Legend

You don't need the .* at the end of the regex, but I like the rest of it. Should match faster and more accurately.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I was thinking the same.

0 Karma

lguinn2
Legend

This probably won't solve the problem, but your timeformat is completely off. It is shown as

 TIME_FORMAT = %m $d %H:%M:%S

should be

 TIME_FORMAT = %Y-%m-%d %H:%M:%S

Plus, you should eliminate the TIME_PREFIX, as it is doing nothing. The LINEBREAKER should also be removed.

All the parsing happens before the transforms. So the settings for the initial sourcetype will be the only setting in effect for parsing: the TIME_FORMAT, etc. does not need to be set on every sourcetype.

Finally, why not set the "default" sourcetype to one of the three, and then you would only need to transform the other two. For example, if you defaulted the sourcetype to SplunkSystem, then you would only need to transform the SplunkActivity and SplunkAudit events. This could be significantly more efficient.

a548506
Path Finder

Lguinn,

The syslog has this in front of what i posted above:

Mar 21 02:00:07 Mar 21 02:00:07 SplunkSystem: 2017-03-20 15:00:06 servername NTP: Synchronized clock via NTP: Successfully slewed time

That is why I have the time format like it is. I did have it like you mentioned, but had no luck getting the transforms to work.

I'll look into setting up to one default like you mentioned.

Thanks

0 Karma

lguinn2
Legend

Then your time settings should be

TIME_FORMAT = %m %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD=19

You might actually get away with eliminating the TIME_FORMAT altogether. But you really do need the MAX_TIMESTAMP_LOOKAHEAD because your events appear to have multiple timestamps.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...