All Apps and Add-ons

Can Splunk process data that is "updated" over time?

mpdude
Explorer

Dear fellow Splunkers,

I have a use case where I believe Splunk could provide great insight, alerts and dashboards, but I do not know if the way data has to be acquired makes it the right tool for the job.

The data in question is timesheet reporting, with the additional challenge that timesheets might be updated (data entry errors fixed) later on.

For example, I could run a script every day that would import records consisting of:

  • ID/Name of the user
  • Current timestamp = the time that data was read from the underlying operational system
  • Timesheet period: Date, begin and end time
  • Project being worked on
  • Maybe additional categories

So, it might happen that I import some of these tuples, but then – say the next day – re-run the import and one of the following happens:

  • A particular period is no longer present, maybe because it has been deleted (time recorded by mistake)
  • A particular period has changed in duration (e. g. forgot to stop timer)
  • New periods are added (forgot to start timer)

Would it be feasible to work with this data in Splunk at all? I guess the problem is that Splunk is not a (relational) database but an append-only index, right? I mean, how could I easily add to all relevant searches that for a particular day, only those events (imported records) are to be considered that have been imported at the time where data for that day has last been updated?

Does that problem description make sense?

Tags (1)
0 Karma

woodcock
Esteemed Legend

This kind of thing is often done using DB Connect:
https://splunkbase.splunk.com/app/2686/
Be aware that v3 is a complete rewrite of v3 and there are many feature changes. If v3 isn't working for you, try v2.

The easiest way to use it is to use dbxquery to access the data in the DB when you need it, but you can also pull it in and index it with Splunk if you like.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

REGISTER NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If ...

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...