Hi,
I have a file in the format /apps/logs/YYYY/MM/DD/system-hostname.log - so, /apps/logs/2014/06/30/system-pf-us123-mgmt.log
My inputs is the following:
[monitor:///apps/logs////system-pf.log]
recursive = Yes
index=network
sourcetype = netscreen_syslog
followTail = 0
disabled = 0
The file is not getting picked up. Did I do something wrong?
Assuming you are trying to regex the date, or the current date of the day (generally, regex does not have access to system time/date however)? I don't see anything that will match the file to the monitor statement either. You could also break this into a monitor statement at the high level (/apps/logs) then dive into the regex with individual whitelist(s).
Perhaps something like (should match a valid date until 2099) :
[monitor:///apps/logs/(19|20)\d\d/(0[1-9]|1[012])/(0[1-9]|[12][0-9]|3[01])/system-\w+-\w+-\w+.log$]
recursive = Yes
index=network
sourcetype = netscreen_syslog
followTail = 0
disabled = 0
--
Kelly
/apps/logs/2014/06/30/system-pf-us123-mgmt.log
/apps/logs/2014/06/30/system-psh-us123.log
/apps/logs/2014/06/30/system-ive123.log
/apps/logs/2014/06/30/system-pf-us299-mgmt.log
and so on, and so on, and so on....
The directory structure will always be the same, and the log file will always begin with system- and end with .log - other than that, it's beyond my control, but it's usually a combination of characters and numbers - sometimes there's an underscore, sometimes a dash...
What does the directory structure look like? What do the system-pf-hosts files inside the dir look like? I was assuming that the files start with "system" then have some series of characters separated by hyphens. Is that not the case? Hence my pattern "system-w+-w+-w+.log$" ... which is system, hyphen, some characters, hyphen, characters, hyphen, charactes, ending with .log ...
In your example above, will "system-pf.log" ever match anything?
Kelly
I want anything under /apps/logs - I control that filesystem, so it shouldn't be a problem. I can't have the system-w+, as that will grab any word, and I have lots of different files in this structure. The pf hosts match one sourcetype, and another name would match a different sourcetype.
So you don't want to validate the date? Careful with '*' ... can be very greedy, and can interfere with regex processing depending on the location. Take a look here: http://docs.splunk.com/Documentation/Splunk/6.1.1/Data/Specifyinputpathswithwildcards
Generally, it's always better to use regex rather than wildcard. With regex you can be much more targeted in what you are after.
Kelly
The asterisk should handle the date - I have plenty of these setup already, and they get picked up.
I would have used
[monitor:///apps/logs/.../system-*.log]
host_regex = /system-([^/]).log$
recursive = Yes
index=network
sourcetype = netscreen_syslog
followTail = 0
disabled = 0
can't be system-*.log, as there are many other inputs that follow the same type of format, based upon the hostname.
yes you are totally correct. Sorry, I've been looking at too many forward and backslashes. Getting cross eyed 😛
no - should be 3 /, no? two for the stanza, one for the actual filesystem.
Hi @a212830
Do you have one extra / before "apps"?