How to retain statistics and metadata while aging out logs?

[devinmcelheran]

Hi everyone,

I think the title sums it up, but I'll clarify anyway.

So, we would like to pull some information from our networking equipment. Very easily done, but we don't want to store it long term. The data we're looking for, at least right now, is primarily the usage statistics, such as what sites people are visiting most, how much traffic is blocked vs traffic that is allowed, etc.

We approached this with Graylog initially, but we severely underestimated just how much logging data we were dealing with. As it turns out, our networking equipment is very verbose. We aren't committed to storing that much data and Graylog doesn't have a straightforward way to store historical statistics and metadata while discarding or aging out the logs through retention policies.

Does Splunk have a way of accomplishing this? If so, would someone mind telling me the Splunk terminology? I've done some searching online, but I can't seem to find what I'm looking for.

Thank you, everyone.

[evania]

Hi @devinmcelheran ,

Did you have a chance to check answers yet? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

[skalliger]

Hi,

not exactly as a feature but with SPL you can do almost everything. And this would be one of the easier steps.

To be exactly, there are two ways (atleast). First one is kinda short: instead of indexing _raw at all, just use indexed extractions to keep the fields you want (a rather rare use case with Splunk).

The other option: summary indexes.
1. Keep in mind that every index got its own retention time.
2. Define indexes that meet your requirements to look further into the past without keeping _raw.
3. Use either collect or mcollect depending on the type of data and store your desired data in separate indexes with longer retention times.
4. Get rid of _raw much faster on the original indexes (reduce retention time).

Skalli