Best way to alert on a new source file for a specific sourcetype?
Bonus point if we can include the parent directory of where the source file is located.
An appliance called "AWS Elemental Live" that encodes and streams video, creates a new directory and a set of logs for every scheduled event. Something like this:
# ls -l /opt/elemental_se/web/log/10000/job_2035/
total 120
-rw-rw-r-- 1 elemental apache 36168 Jan 28 00:55 20200128T005500_emecmd.xml
-rw-rw-r-- 1 elemental apache 44368 Jan 28 02:05 20200128T005501_eme.log
-rw-rw-r-- 1 elemental apache 32955 Jan 28 02:05 20200128T005501_eme_ve.log
-rw-rw-r-- 1 elemental apache 2618 Jan 28 02:04 20200128T005501_ingest_2046.log
How do I create an alert whenever that happens?
The alert ideally should be something like this:
Splunk is reporting that "job_2035" has started recording on $host$
... (then I could try to also alert on the end of the recording if I could figure out how)
(I haven't found unique or common enough events in those logs - "recording started" or some such - so I could alert on those events rather than on the source file creation. So the alert condition must be the fact of a new source file creation.)
Thanks!
You might also tinker with the transaction command using startswith and endswith
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction
you can optionally add a field extraction for your Job id like so
| rex field=source "/(?<job_id>job_\d+)/"
This will look for events where first event time is under 10 minutes. You can adjust the threshold age_seconds below up/down to your liking. Adjust the index and sourcetype constraints to match your actual index and sourcetype.
| tstats min(_time) as first_time where index="*" sourcetype="*" source="*job_*" by sourcetype source
| fields first_time index sourcetype source
| eval current_time=now()
| eval age_seconds=current_time-first_time
| search age_seconds<=600
| eval age=tostring(age_seconds,"duration")
| eval age = replace(age,"(\d+)\:(\d+)\:(\d+)","\1h \2min \3s")
| convert ctime(first_time) ctime(current_time)
| table current_time first_time age age_seconds sourcetype source