Evaluating if splunk is for me.

[infinitiguy]

Hi Everyone,

I'm trying to find a log solution and here is what I would like to achieve.

• I have 50 systems with weekly messages aggregation of under 500MB a week.
• I also have jboss applications running on the same 50 nodes that I'd like to capture their error.log's (but not server.log).
  
• I also want to filter what actually gets sent to splunk as I'm only interested in the first line of the stacktraces.
  

I can filter these out using egrep for a date format - which brings my 100M log down to 4M. Does splunk have any capability to do filtering before it actually brings something in to index? Sometimes our logs can get out of control and I can write 2-5GB of error.logs within a couple hours - most of which I'm not interested in, and wouldn't want in splunk, which would cause me to go over the 500MB free threshold.

Anyone have any thoughts? How do other people handle similar types of problems?

[dmurphy@jboss11 ~]$ egrep [0-9]{4}-[0-9]{2}-[0-9]{2} error.log.1  
2012-01-22 13:02:36,548 [http-0.0.0.0-8080-223] ERROR [org.apache.commons.beanutils.PropertyUtils] Method invocation failed.
2012-01-22 13:04:08,114 [http-0.0.0.0-8080-105] ERROR [org.apache.commons.beanutils.PropertyUtils] Method invocation failed.

[RicoSuave]

The short answer is YES. Everything that you are looking to do can be done with splunk. I won't go into the details because you are better off reading the documentation and playing with splunk yourself, but it's not hard at all to configure splunk for your requirements. My recommendation is to download splunk, and go through the tutorials available in the documentation. Then read the sections that deal with installing and administration of splunk. And of course, once you have more detailed questions, with regards to configuration, ask them here.

[wwhitener]

For what it's worth, I'd also think about doing a support contract for a short while. Then you get some expert help when something particularly tricky shows up.