I've seen the documentation and believe there is a way to dynamically do this with props.conf but I'm not understanding how to do it. I my case I'm working with 15 different source types with different file names, but at the same nested directory level.
Only one works at a time, but if both are enabled, only the last one works. Both stanzas below are similar but one has disktool.txt and one has diskview.txt.
inputs.conf
[monitor://\\host.share.comUploadDatasupportdata_Customers...*.disktool.txt] crcSalt = <source> index = eql_disktool sourcetype = disktool
[monitor://\\host.share.comUploadDatasupportdata_Customers...*.diskview.txt] crcSalt = <source> index = eql_diskview sourcetype = diskview
Thanks,
Rob
I would recommend an approach similar to this:
(inputs.conf on the forwarder)
[monitor://\\host.share.comUploadDatasupportdata_Customers]
whitelist = disk(view|tool)\.txt$
(props.conf on the forwarder & indexer)
[source::...diskview.txt]
sourcetype=diskview
[source:...disktool.txt]
sourcetype=disktool
[diskview]
TRANSFORMS-index = diskview-index
[disktool]
TRASNFORMS-index = disktool-index
(transforms.conf on the indexer)
[diskview-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = diskview
[disktool-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = disktool
This avoids have overlapping (or nearly overlapping) monitor stanzas, and sets the sourcetype of each file by name. Once the sourcetype is set, it uses index-time transforms to move the data into the correct indexes.
Wow - a million files is definitely a performance problem. Are all the files "live" or are some of them stale? Check out some of the inputs.conf
settngs - or better yet, move stale files to another directory after some appropriate time lapse (like a week).
I'm now thinking this may be just a performance issue since a single indexer is trying to ingest more than a million files. It may be just working through one rule at a time. That would make sense why each rule works individually.
Thanks, will let you know
Try it without the crcsalt, and see if you get my results. I have not used that yet, because it is bad juju.
For me, I get the same behavior on my local laptop with no share. Doesn't seem to like the combination of wilcard ... and a similar path. If I disable the last source, the next to last source starting indexing events 🙂
There is definitely something wrong with shares. I cannot get this to break on local drives. I'll test it on shares tomorrow.
Rule works as long as you only have one monitor stanza active otherwise it seems to conflict with others.