Getting Data In

using splunk for debug or trace logs

aviadr1
Explorer

I have a complex system which sometimes needs to be debugged or troubleshooted by using verbose trace logs. the challenges in my scenario are:

  1. the verbose logs grow fast, in the order to 5Mb - 20mb per minute.
  2. the machines producing the logs are also cpu constrained, as they run cpu intensive tasks
  3. the machines producing the logs are located at a remote client location (we have many clients), these are far(bandwidth is limited) from the location where we host the main splunk indexer

what would be a good topology in this case?

0 Karma
1 Solution

Genti
Splunk Employee
Splunk Employee

This can be one way of dealing with this; i must admit, you seem quite limited in resources but:

  • i dont think 1 would be an issue if 2 and 3 were not around. (i would guess that these logs will only grow so large at very rare occasions and not constantly and all time)
  • From 2, it seems like you should run LightWeightForwarders which have the lowest footprint and CPU usage
  • From 3, it seems like you are being bandwith limited and cant really send the data to the main indexer which is in a different location.(that would be the ideal option)

If it is possible to have an extra box dedicated to splunk in the remote location, then you can install splunk as an indexer (at this remote location), collect the remote logs into this indexer and then allow the main splunk indexer to search the remote one, (basically adding search peers).

View solution in original post

Genti
Splunk Employee
Splunk Employee

This can be one way of dealing with this; i must admit, you seem quite limited in resources but:

  • i dont think 1 would be an issue if 2 and 3 were not around. (i would guess that these logs will only grow so large at very rare occasions and not constantly and all time)
  • From 2, it seems like you should run LightWeightForwarders which have the lowest footprint and CPU usage
  • From 3, it seems like you are being bandwith limited and cant really send the data to the main indexer which is in a different location.(that would be the ideal option)

If it is possible to have an extra box dedicated to splunk in the remote location, then you can install splunk as an indexer (at this remote location), collect the remote logs into this indexer and then allow the main splunk indexer to search the remote one, (basically adding search peers).

aviadr1
Explorer

I'm thinking splunk isn't built to handle debug level logging

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...