Getting Data In

Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed
Explorer

I am trying to setup Splunk to monitor a remote tomcat instance ( catalina.out ) for messages like permGen Running out of Memory
Specifically:

Exception in thread "http-bio-8080-exec-36" java.lang.OutOfMemoryError: PermGen space

I was able to install Splunk on host A, and on B i have Tomcat running, plys Universal forwarder running with:

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf

[monitor:///usr/share/apache-tomcat-7.0.47/logs]
sourcetype = access_common

/opt/splunkforwarder/etc/system/local/outputs.conf

forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=<server where splunk server is installed>:9997

So how do i :
1. Make sure the forwarder HAS Connectivity and is able send logs, some command command line utilities perhaps
2. How do i setup the receiver / splunk server ?

Tags (1)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

kamal2222ahmed
Explorer

and this Works!, thanks basic config is SO simple in Splunk, quite amazing. I wish the documentation was more use case driven.
next .....:

  1. extract , or plot only the PermGen log
  2. Setup Notifications ( Email ) upon occurrence of Error
  3. Setup another log parser to get application errors
  4. Correlate the two errors temporally
0 Karma

kamal2222ahmed
Explorer

so the username and password for the command :
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
are local ? meaning, i can choose the password for user splunk, which would be local the forwarder ?
ok i used admin:changeme

/opt/splunkforwarder/bin/splunk add forward-server vm-jenkins-staging.3mhis.vm:9997
Splunk username: admin
Password:
Added forwarding to: vm-staging.vm:9997.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The forwarder has no clue about your indexer's credentials, use admin:changeme on the forwarder.

0 Karma

kamal2222ahmed
Explorer

@martin_mueller
more /opt/splunkforwarder/etc/system/local/outputs.conf
forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=vm-staging.vm:9997

0 Karma

kamal2222ahmed
Explorer

I tried to run add forward-server on the forwarder , with the same admin credentials as i use to login to the indexer, but getting error:
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
Splunk username: admin
Password:
Login failed

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...