it should be fine as long as you get your required order proper and make sure not to overwrite anything that would be changed during normal operations of splunk. I'm planning on writing a splunk module to deploy universal forwarders and I don't see any reason why it con't be done. ... View more

this was exactly what I was looking for ... View more

do i need to use span to be able to set the height of my chart? I edited the xml and tried adding in the charting.chart.height but my chart would no longer draw. I tried from 500 up to 4500 but still no luck ... View more

I don't believe so. I asked a similar question in the #splunk irc channel and I don't think we can do 3d. I had wanted to do syslog processes by host over time. Instead I just created multiple charts with defined timeframes ... View more

exactly what I was looking for! ... View more

Is it possible that rsyslog is sending stuff over in a way that splunk doesn't know how it should be tagged so it's not doing any stripping? Doesn't seem right because both are appearing under sourcetype=syslog. ... View more

hrm - not so sure that is it. I just tried every format listed on that page - except for debug, and the timestamps never changed in splunk - still getting the duplicates. ... View more

spot on. So now the question is, which format do I want? I'm thinking either RSYSLOG_FileFormat or RSYSLOG_ForwardFormat? Do you know which will give me entries similar to the below in splunk? Feb 13 17:22:21 syslog5client.domain.com sshd[13812]: pam_unix(sshd:session): session closed for user dmurphy ... View more

posting as an answer because pasting is bad in a comment. Essentially I want to take the below input 2012-02-10 13:04:51,208 [http-0.0.0.0-8080-252] ERROR Rejecting request due 2012-02-10 13:04:51,348 [http-0.0.0.0-8080-29] ERROR [NOTIFY]: some error lots of stuff fdsfasdkfljdskfsadjd fsadjklfjsdaklfasdjf fsjadklfjsdalkfjdsaklfjsd 2012-02-10 13:05:06,895 [http-0.0.0.0-8080-298] ERROR [org.apache.commons.beanutils.PropertyUtils] Method invocation failed. java.lang.IllegalArgumentException: java.lang.ClassCastException@73afa6d1 at sun.reflect.GeneratedMethodAccessor275.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.beanutils.PropertyUtilsBean.invokeMethod(PropertyUtilsBean.java:1773) 2012-02-10 13:04:45,873 [http-0.0.0.0-8080-151] ERROR [site.mismatch.DupDetection] Rejecting request due 2012-02-10 13:04:51,208 [http-0.0.0.0-8080-252] ERROR [site.mismatch.DupDetection] Rejecting request due and I want to break it into the following events - for clarification I'll separate with ########## 2012-02-10 13:04:51,208 [http-0.0.0.0-8080-252] ERROR Rejecting request due ########## 2012-02-10 13:04:51,348 [http-0.0.0.0-8080-29] ERROR [NOTIFY]: some error lots of stuff fdsfasdkfljdskfsadjd fsadjklfjsdaklfasdjf fsjadklfjsdalkfjdsaklfjsd ########## 2012-02-10 13:05:06,895 [http-0.0.0.0-8080-298] ERROR [org.apache.commons.beanutils.PropertyUtils] Method invocation failed. java.lang.IllegalArgumentException: java.lang.ClassCastException@73afa6d1 ########## at sun.reflect.GeneratedMethodAccessor275.invoke(Unknown Source) ########## at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ########## at java.lang.reflect.Method.invoke(Method.java:597) ########## at org.apache.commons.beanutils.PropertyUtilsBean.invokeMethod(PropertyUtilsBean.java:1773) ########## 2012-02-10 13:04:45,873 [http-0.0.0.0-8080-151] ERROR [site.mismatch.DupDetection] Rejecting request due ########## reason being is I then want to match any line starting with ^\s and throw it away (getting rid of the java stack trace), and it looks like you can only throw away whole events, not partial events. Technically the 4 "at" lines were part of the IllegalArgumentException that preceded it, but I don't want it because in real life, there's 80 more "at" lines that follow and they are not interesting to me. ... View more

the one thing I'm really missing is the ability to match on a string(^20), and say all lines until the next match belong to this, except for lines that start with a space - if space, then break into new events UNTIL next match of ^20, which is in itself a new event I can't seem to get splunk to do something like this. ... View more

yep - I've been using nullQueue, which has solved half the problem, but there's still the issue of all my hosts sending all those logs over in full format to get chomped by the indexer where it would be more efficient if the individual nodes could do the trimming before sending over. ... View more

The reason I'm interested in a heavy forwarder is to reduce the noise that gets sent to the indexer. Not a big deal with one server, but if we deploy to dozens of servers and something goes wrong and the logs start spewing GB's of log entries when we're only interested in a couple lines in each java stack trace (what I'm monitoring) it'll be very easy to get into a license violation, as well as a lot of extra network traffic that I can reduce with a little bit of splunk and cpu muscle. I was able to parse things great with logstash, but it's a bit of a hacked solution and not as elegant as splunk. I've yet been unable to get the splunk forwarder to parse the data the way I want so the hope is to get that working so I can use a entire splunk solution. ... View more

also - to clarify. Much of my confusion was around the fact that nullQueue is for sending entire events to the nullQueue - not part of a single event, which I wasn't clear about. Once I was able to split my stack trace into separate events it became clear how to send some to null. With a little more work I think I can get single events that don't contain noise. ... View more

is it possible to BREAK_ONLY_BEFORE_DATE and MAX_EVENTS = ? I'm looking for a way to truncate jboss error logs (full of java exceptions). ... View more

I tried the props and transforms suggestions but I can't seem to get them to work. I still end up with the whole stacktrace being indexed 😞 ... View more

spoke too soon. It looks like MAX_EVENTS = 5 is taking my 87 line log entry and chopping it up into blocks of 5 instead of truncating. I just added another server that had a far noiser error.log (100MB worth) and it looks like i'm getting events that are pieces of a java exception. ... View more

I think I found something that works. client: inputs.conf [default] host = n1a901 [monitor:///opt/jboss/server/default/log/error.log*] sourcetype = med_log4j server: props.conf [med_log4j] MAX_EVENTS = 5 this results in search output that only has 5 lines - compared to the 87 I normally see. I assume it's only the 5 because I no longer see "show all 87 lines". Can anyone confirm that MAX_EVENTS is actually truncating the lines and not indexing them and that it's not just some sort of gui filter? 🙂 ... View more

thinking about it more I think I can go about this a different way - not completely sure how yet. I setup a udp port (514) for syslog and the only other logs im interested in are the jboss logs which will be read locally and sent via a forwarder. The first 5 lines of each "event" are the only ones I'm interested in. Typical jboss events seem to have 80+ lines. I tried adding TRUNCATE=128 to props.conf but that limited each of the 87 lines to 128 characters. I was hoping it would limit the event itself. Is there an easy way to say I don't care what the event is, please just limit it to 5 lines of data? ... View more

yes - the jboss error.log is full of java stack traces. Currently I have the clients set up with an inputs.conf of: [dmurphy@n1a901 local]$ sudo cat inputs.conf enter code here [default] host = n1a901 [monitor:///var/log/messages*] sourcetype = syslog [monitor:///opt/jboss/server/default/log/error.log*] sourcetype = jb_log4j outputs.conf looks like: [dmurphy@n1a901 local]$ sudo cat outputs.conf [tcpout] autoLB = true maxQueueSize = 500KB forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = _audit forwardedindex.filter.disable = false indexAndForward = false autoLBFrequency = 30 blockOnCloning = true compressed = false disabled = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 heartbeatFrequency = 30 maxFailuresPerInterval = 2 secsInFailureInterval = 1 maxConnectionsPerIndexer = 2 sendCookedData = true connectionTimeout = 20 readTimeout = 300 writeTimeout = 300 useACK = false defaultGroup = n1s901.domain.com_9009 [tcpout:n1s901.domain.com_9009] server = n1s901.domain.com:9009 [tcpout-server://n1s901.domain.com:9009] Since the files are locally on the box I'm hoping to be able to have splunk look at sourcetype for different filtering. syslog I don't want filtered at all, but jb_log4j sourcetype I do want filtered. Instead of [source::/opt/jboss/default/logs/error.log] could I do something like [source::sourcetype=jb_log4j] The end result we want to work towards is having splunk be "the" syslog server. Can splunk be configured to directly listen for syslog traffic? i.e. if I have splunk listening on 514 - can I have syslog just forward to the splunk IP? I want to avoid having to have forwarders on all of my hosts for just syslog traffic. The jboss hosts (only 3) will have forwarders. I also don't want to have syslog writing events to a local messages file on the splunk server - just for splunk to read. If it has to be setup that way then that's ok - but I'm hoping to just have native syslog send over the network direct to splunk. Is that possible? ... View more