Getting Data In

Where can I find information on how to track my indexing volume over time?

mctester
Communicator

I am trying to find a search for index volume over time for licensing tracking.

I need to search by index or by host or by source type.

I need to provide a line chart to show growth and projected license violation to show when we will need more licensing and were we could provide filtering to lower event collection.

I also need to learn how to do event filtering from the client to lower unnecessary event collection.

1 Solution

Mick
Splunk Employee
Splunk Employee

You can find both guidance and searches to investigate your indexed data volume here. There are a number of searches on that page that will break down your indexed volume by host, source, sourcetype and index over time. By editing the search, you can look at weekly, daily or hourly totals, depending on your needs.

As for filtering events to keep your volumes down, that is all explained in our documentation here. This is resource intensive and will impact the speed at which your Splunk instance can index data, so if possible I would implement any filtering at the app level, rather than relying on splunk to do it

View solution in original post

hartfoml
Motivator

Thanks Mick, I tried your recomendation and pulled up only parcal ansers. For some reason the _interanl records for each day were stored under only a few dates like this -- 9/29/10 12:01:15.000 AM 10-04-2010 00:03:57.890 INFO LicenseManager-Audit - Audit:[quotaExceededCount=1, lastExceedDate=1284181219, peak=26631181931, rolloverCount=116, totalCumulativeBytesAtRollover=780873525059, todaysBytesIndexed=179325960, licenseSize=10240][MCp8taTlG+OTQsYggKfV0oVaeoCO9dPKPEYgWaOsf8qw6YpLbJjsgwTXmASiPbv65YE662cFMxu4UNySTNzL1FDZR6AdO/YBN9SgRw/u4TJOfsWF9gBSaOjnFYRGa7qR8ZpzCO3nJtEP7XAA9xSz0ScCQDRpRVHVwgtvhPbXwQn9WOaVy5rmoZKtKn/RbkmauPpEJPQulBHOd+l5RXaI26Ej0JX+qt9tdLFP7wjhHqjv6+CwdXmvdl1yZTWDvqeXKVNDFvl0+OJ8raLe8hwLbJTNoI/1igCnE+2mCKKOirkvtR9b6Jg1HX6n8Mg+vYwA3k4b5YzVCkA9cMPVbE9Egw==] host=escman01 Options| sourcetype=splunkd Options| source=D:\SPLUNK\var\log\splunk\license_audit.log Options

this record was created on 10-4 but the timestamp on this record is 9/27/10

this makes it hard to search since several records are timestamped with the same date and only one record is showen

0 Karma

Mick
Splunk Employee
Splunk Employee

You can find both guidance and searches to investigate your indexed data volume here. There are a number of searches on that page that will break down your indexed volume by host, source, sourcetype and index over time. By editing the search, you can look at weekly, daily or hourly totals, depending on your needs.

As for filtering events to keep your volumes down, that is all explained in our documentation here. This is resource intensive and will impact the speed at which your Splunk instance can index data, so if possible I would implement any filtering at the app level, rather than relying on splunk to do it

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...