Splunk Search

Control number of sources with rotated logfiles

Starlette
Contributor

I am monitoring a dir with rotating logs, ( fi /depot/logs/ ) how can I control the source name, and avoid zillions of sources. (file_1.log file_2.log)

thanks! Starlette

Lowell
Super Champion

Just FYI, I've posted several fully functional source renaming transformers in another answer. (This is using the approach that southeringtonp is talking about.) Feel free to take a look and see if any of them will work for you: (Specifically, the transformer name "source_clean-digits-before-ext" looks like it will work for your situation.)

0 Karma

southeringtonp
Motivator

In inputs.conf, you can explicitly set the value of source for a given input definition:

[monitor:///var/log/something]
disabled = false
sourcetype = mysourcetype
source = mysource

Or, you can use a transform to assign it in a more targeted way:

[mysourcetype]
DEST_KEY = MetaData:Source
REGEX = (?=)
FORMAT = source::mysource

The above example will always set the source - adjust the REGEX setting as needed to match text in your events for a more targeted assignment.

Starlette
Contributor

ah this looks promising,,,thanks!

0 Karma

williamche
Path Finder

You could try the following in your props.conf file to specify a sourcetype based on the file's naming convention:

[source::/depot/logs/file_*.log]
sourcetype = foo
0 Karma

williamche
Path Finder

Ah, I see what you did there! I must've read too much into Starlette's questions and thought that all the data from each log file were assigned to a unique sourcetype named after the filename. It happened to me when I left the sourcetype = automatic. So I used the method I suggested to overwrite the sourcetype so they are the same for all the rotated log files. (-2.. I have to make that up somehow! 🙂 )

0 Karma

southeringtonp
Motivator

This sets sourcetype, not source.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...