Getting Data In

Mulitple versions of logs creating separate sourcetypes?

onegreydot
Explorer

I have a log file that I created a transforms.conf and props.conf for specifying the log source in the props with [source::.../name.log]. The application generating the log was recently updated and the log file now has the same name.log but different header fields. Righ now I am extracting the fields using the REPORT statement and having DELIMS and FIELDS. But since the log file changed the FIELDS are now different. What would be the best way to maintaining both versions of logs inside of my splunk instance without having to rename the actual log?

0 Karma

emiller42
Motivator

Do the existing transforms still function on the new format of data, just with incorrect fields? Or does it not apply at all?

If it doesn't apply at all, then you can simply add another transforms.conf stanza for the new format, and have the props.conf stanza use both. Splunk will try both and apply whichever works.

However, if your extraction still technically works, just is incorrect because the data changed, that complicates things. Can you provide sample data and your existing configuration?

0 Karma

emiller42
Motivator

You may have to get more explicit and use regex to parse out the fields then. I would try to make regex extractions that only match one of the two formats. That way when both are applied, there's no way the fields can be swapped around like that.

0 Karma

onegreydot
Explorer

Tried adding that in the props.conf and it does extract both sets now.

However it still doesn't distinguish between logs, because the headers have changed. I should have put that in the example above.

So instead of "src_ip", "src_port", "dest_ip" ...
It may be "src_ip", "src_port", "host", "dest_ip" ...

The above makes it so that host could have the dest_ip and vice versa when searching the logs. If there were just additional headers to the log the above solution would work great!

Thanks!

0 Karma

emiller42
Motivator

Try just this in your props.conf settings:

REPORT_name_log = old_log,new_log
0 Karma

onegreydot
Explorer

The format of the log files in the transforms.conf looks something like:
old log:

[old_log] FIELDS="timestamp", "src_ip", "src_port", "dest_ip", "dest_port"
DELIMS=","

new log:
[new_log]
FIELDS="timestamp", "src_ip", "src_port", "dest_ip", "dest_port", "url", "level", "message"
DELIMS=","

In my props.conf I have:
[source::.../name.log]
sourcetype = name_log

[name_log]
REPORT-old_log = old_log
...
...

As you can see the new version of the log has a few more fields. I will actually try putting both REPORT statments in the props.conf and see how that goes.

  • Thanks for the help.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...