I know there are similar questions, but not exactly and the answers don't seem to apply. Also, I'm a noob so forgive me if my terminology isn't exactly correct.
I have a forwarder running on Server1 and an indexer on Server2.
My inputs.conf is simple and looks like:
[monitor:///opt/mypath/*.log.2011-04-*]
It looks like that because I'm evaluating Splunk and to keep below 500MB I can only index this month's log files.
my files are log4j type outputs and have a naming convention as such:
source-version.log.yyyy-mm-dd
where source = name of software producing the file, version = release version of the source (ex. 1.8.1, 1.10, etc). The rest, I'm sure, is self explanatory.
The behavior I'm seeing is that some files are being indexed, and others are not. I can't find a consistent pattern.
for instance:
some, but not all of sourceA files are being indexed.
1 of sourceB files are being indexed
none of sourceC files are being indexed
I added crcSalt=<SOURCE> to my inputs.conf file, and it resulted in a small increase in files being read (which came through as a single event, another problem) but I'm still missing nearly half of them.
I've looked in splunkd.log (on both servers) but didn't see anything referencing the missing file names.
Some background:
orginally I tried indexing the entire directory, but it quickly breached 500MB, so I had to do a ./splunk clean eventdata .
also, I created my own datetime.xml so I can derive the date from the filename, which I reference in a local props.conf file.
datetime.xml:
<datetime>
<define name="_fndate" extract="year, month, day">
<text><![CDATA[source::.*?\\*.log.\d{4}-\d{2}-\d{2}]]></text>
</define>
</datetime>
props.conf:
[my_sourcetype]
DATETIME_CONFIG = datetime.xml
It seems unlikely that this would cause problems, as other files with the same naming convention, and in some cases same log format, get indexed.
Also, if I get rid of both props.conf and datetime.xml from my local directory and restart the forwarder there is no change.
Thanks for reading!
... View more