Getting Data In

Inputs.conf help

plumainwfs
New Member

I am trying to onboard ingest about 30 different log type from a single Source (Linux Server)

Currently the logs are being written and zipped by rsyslog in a particular folder (structure below):
/mnt/log/files/YYYY/MM/DD/hostname/filename.log.gz

What would be the best way to ingest the different logtypes

I was thinking about the following:

[monitor:///mnt/log/files/////]
host_segment=7
whitelist=(.+\access.gz)
index=webserver
sourcetype=access:apache

Not sure why the asterisks are not showing up on the preview;
it should be /mnt/log/files/aster*/aster*/aster*/aster*/

I was also thinking to ingest all 30 log types I would need to create 30 different configs?

0 Karma

lguinn2
Legend

If the different log types are going to different indexes, then yes, you will need 30 different configs. But if the only difference will be the sourcetype, you could do an override in props.conf. You still have to configure all the different configs, but it might be easier to do. Take a look at this:

inputs.conf
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webserver

props.conf
[source::/mnt/log/files/.../access.gz]
sourcetype=access:apache

[source::/mnt/log/files/.../otherthing.gz]
sourcetype=st_otherthing

etc.

0 Karma

plumainwfs
New Member

It would most probably be about 4-5 different indexes
Best way to do this would be with the inputs as so..
inputs.conf
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webserver

[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webapp

All 30 log types would be defined as separate sourcetype
so best way do to this would be with the props.conf as you outlined?
props.conf
[source::/mnt/log/files/.../access.gz]
sourcetype=access:apache

[source::/mnt/log/files/.../error.gz]
sourcetype=st_otherthing

0 Karma

plumainwfs
New Member

Also what happens to a gz file that I am not aware of? will that get indexed? and will not have a pre-defined sourcetype?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...