Getting Data In

Managing rotated syslog files

francoisjoannet
New Member

Hi there, simple question but I can't get my head around this.

I've got a hosts that manages it's logging with syslog. Files are been rotated regularly. The folder containing the files is been sent with local light forwarder to the indexer.

I've got files hierarchy like:

/var/log/system.log

/.../.../system.1

/.../.../system.2

/.../.../system.3

/var/log/named.log

/.../.../named.1.log.gz

/.../.../named.2.log.gz

...

You get the point. There is more than hundred files and some of those files will get removed after some time or X amount of file, or just over specific size.

/var/log/ is set as my source on that host and white listed *.log and *.log.*

Now it's kind of a mess in my indexer as there are so many files now.

Questions:

How can I clean this mess my kinda merging all the files named.*.log.gz under named.log? How can I better configure my forwarder to streamline those sources in a more convenient way? And last, would it make it different if I use a forwarder instead or a light forwarder?

Thanks,

cheers all.

Tags (2)
0 Karma

Lowell
Super Champion

Splunk shouldn't have a problem monitoring a few hundred (or more) files like this. One word of caution though: I would highly recommend that you ensure that splunk is NOT indexing any of your *.gz files. This is because splunk will normally handle rotated files automatically, but as soon as a file is gziped, it thinks it's a new file and will therefore index it again. Also, if you're using a tool like the unix logrotate utility, then I suggesting making sure that you have the "delaycompress" feature enabled. This will keep the file from being compressed immediately, giving splunk a slightly bigger window in which to get all of your events (Just in case splunk was down at the exact moment you file was rotated and compressed. It's not likely, but I prefer trying to reasonably cover all the angles.)

What I've found to work the best it to simply setup a source-rewriting transformer for the sourcetype(s) affected. I've posted about this in some detail before, so instead of duplicating it all here, I'll simply point you to that answer:

You'll probably find source_clean-trailing-digits and source_clean-digits-before-ext to be the most helpful.

And yes, if you are using a forwarder then the transformations happen locally before the data is forwarded to the indexer (and the config files are put on the forwarder). If you are using a light-weight forwarder then the config settings will have to go on the indexer. I use the heavy-weight forwarders, fyi.

Genti
Splunk Employee
Splunk Employee

so my question is, what seems to be the problem? You have a lot of files, and so you have a lot of sources. I understand that. But why is this a problem for you, currently?

If you would like to organize the files better you could set a specific sourcetype for all of them, and so for example for all the named.log files you could have a "namedlog" sourcetype and so "gather" them this way.

You could setup eventtypes as well..etc.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...