Getting Data In

job based log file processing

levinro
Engager

Newbie here - Just evaluating Splunk.

I set up my source to watch a directory and my source type filtering by file name pattern. This is all great, it parsed out errors & messages nicely with no effort.

Now I want to take it a step further. When each file arrives and is closed (the discrete job finished), I want to evaluate things like run time (time between last & first time stamp in the log file) and record counts (a string pulled from a specific message in the log file). Then I want to create trending reports on these values on a daily/weekly/monthly basis.

Tags (4)

jtacy
Builder

Using metadata is probably the fastest way to get the start/end timestamps for each source. You can then get the difference in seconds pretty easily. You'll need to have some kind of search that returns the record count for each file and that might involve using rex to pull that field out. That field extraction should happen in the subsearch part of the example below:

| metadata type=sources index=yourindex | eval duration=lastTime-firstTime | join source [search index=yourindex recordcount=* | top 1 recordcount by source] | table source, duration, recordcount

If you're going to have a lot of events, optimizing that subsearch will be crucial when you start getting into those weekly/monthly groupings. There will be some stuff to add to turn this into a chart, but I'd probably start with the field extraction and getting the duration and record count into a table and go from there. Good luck!

0 Karma

lukejadamec
Super Champion

Individual log files are generally considered to be a Source, not sure if you can do a transaction on a Source.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...