Greetings Splunkers!
I am currently collecting logs centrally for a content delivery platform for indexing into Splunk.
The vendor in their infinite wisdom has decided that the service_monitor
logs should use the same file naming convention, despite the format of the logs differing based on the type of device that is generating it.
Example:
Filename: service_monitor_10.10.18.49_20110920_204501_00363
Originating Device Type: Service Engine
Fields: date time movie-streamer-threshold-exceeded movie-streamer-augment-threshold-exceeded movie-streamer-stopped...
Filename: service_monitor_10.10.18.41_20111026_225501_03295
Originating Device Type: Service Router
Fields: date time sr-cpu-percentage sr-mem(bytes) requests-received http-normal-requests-received...
Keeping in mind that these files are kept in the same directory. It would no doubt be possible to determine the role by the IP address, however this would involve a LARGE inputs.conf
with a stanza something like:
[host://<ip_address>_service_monitor'
sourcetype=service_monitor_se
...
For every device on the platform.
Is there a way I can differentiate between the two automatically?
Many thanks in advance 🙂
RT
Playing around has come up with the goods.
With a list of the Service Routers, we were able to come up with the following stanza in our inputs.conf
:
[batch:///...path_to_file.../service_monitor_(10.10.10.163_|192.168.159.68_|10.10.10.172_|192.168.159.76_|10.10.0.76_|192.168.159.172_|10.10.10.67_|192.168.159.164_)*]
host_regex = service_monitor_(\d+.\d+.\d+.\d+)_\d+_\d+_\d+
sourcetype = service_monitor_sr
index = service_monitor_sr
crcSalt = <SOURCE>
disabled = false
move_policy = sinkhole
Because configuration files are processed sequentially, having a "catch-all" below it for the other files captures the rest of them as service_monitor_se
sourcetypes:
[batch:///...path_to_file.../service_monitor*]
host_regex = service_monitor_(\d+.\d+.\d+.\d+)_\d+_\d+_\d+
sourcetype = cds_service_monitor_engine
index = cds_service_monitor_engine
crcSalt = <SOURCE>
disabled = false
move_policy = sinkhole
I hope this helps someone 🙂
Playing around has come up with the goods.
With a list of the Service Routers, we were able to come up with the following stanza in our inputs.conf
:
[batch:///...path_to_file.../service_monitor_(10.10.10.163_|192.168.159.68_|10.10.10.172_|192.168.159.76_|10.10.0.76_|192.168.159.172_|10.10.10.67_|192.168.159.164_)*]
host_regex = service_monitor_(\d+.\d+.\d+.\d+)_\d+_\d+_\d+
sourcetype = service_monitor_sr
index = service_monitor_sr
crcSalt = <SOURCE>
disabled = false
move_policy = sinkhole
Because configuration files are processed sequentially, having a "catch-all" below it for the other files captures the rest of them as service_monitor_se
sourcetypes:
[batch:///...path_to_file.../service_monitor*]
host_regex = service_monitor_(\d+.\d+.\d+.\d+)_\d+_\d+_\d+
sourcetype = cds_service_monitor_engine
index = cds_service_monitor_engine
crcSalt = <SOURCE>
disabled = false
move_policy = sinkhole
I hope this helps someone 🙂
If you extract each hosts by using a regular expression, does it work for you? The setting will be following in inputs.conf.
[monitor://
host_regex = $YOUR_REGEX
Thanks Takajian. I can see what you tried to do there, but it's not quite what I was after. Fortunately I have found an swer that does what I need it to do... see below 🙂
Thanks again for your answer!