I am trying to onboard ingest about 30 different log type from a single Source (Linux Server)
Currently the logs are being written and zipped by rsyslog in a particular folder (structure below):
/mnt/log/files/YYYY/MM/DD/hostname/filename.log.gz
What would be the best way to ingest the different logtypes
I was thinking about the following:
[monitor:///mnt/log/files/////]
host_segment=7
whitelist=(.+\access.gz)
index=webserver
sourcetype=access:apache
Not sure why the asterisks are not showing up on the preview;
it should be /mnt/log/files/aster*/aster*/aster*/aster*/
I was also thinking to ingest all 30 log types I would need to create 30 different configs?
If the different log types are going to different indexes, then yes, you will need 30 different configs. But if the only difference will be the sourcetype, you could do an override in props.conf. You still have to configure all the different configs, but it might be easier to do. Take a look at this:
inputs.conf
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webserver
props.conf
[source::/mnt/log/files/.../access.gz]
sourcetype=access:apache
[source::/mnt/log/files/.../otherthing.gz]
sourcetype=st_otherthing
etc.
It would most probably be about 4-5 different indexes
Best way to do this would be with the inputs as so..
inputs.conf
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webserver
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webapp
All 30 log types would be defined as separate sourcetype
so best way do to this would be with the props.conf as you outlined?
props.conf
[source::/mnt/log/files/.../access.gz]
sourcetype=access:apache
[source::/mnt/log/files/.../error.gz]
sourcetype=st_otherthing
Also what happens to a gz file that I am not aware of? will that get indexed? and will not have a pre-defined sourcetype?