Splunk Search

how can I troubleshoot logs not getting forwarded to indexers

maxruas
New Member

Hi,

I have splunkforwarder installed to monitor garbage collection for glassfish servers A and B side.
I have a dashboard on splunk for that and some of it stopped working "Real time GC pauses by type" as logs stopped coming in for this.
every thing looks correctly configured and looks same as the B side that works.

Inputs.conf:

[monitor:///zones/*/root/opt/SUNWappserver/glassfish/domains/scdomain/logs/gc.log*]
sourcetype=glassfish:gc
blacklist=(.*\.gz$|.*\.xml$|.*\.1\d*$|.*\.2\d*$|.*\.3\d*$|.*\.4\d*$|.*\.5\d*$|.*\.6\d*$|.*\.7\d*$|.*\.8\d*$|.*\.9\d*$)
whitelist = (.*\.log$|.*\.*log.0)
ignoreOlderThan = 30m
index=nld_cdxx_app
host_segment = 2

[monitor:///zones/*/root/opt/SUNWappserver/glassfish/domains/scdomain/logs/access/*.txt]
sourcetype=glassfish:access
blacklist=(.*\.gz$|.*\.xml$|.*\.1\d*$|.*\.2\d*$|.*\.3\d*$|.*\.4\d*$|.*\.5\d*$|.*\.6\d*$|.*\.7\d*$|.*\.8\d*$|.*\.9\d*$)
whitelist = (.*\.txt$)
index=nld_cdxx_app
host_segment = 2
ignoreOlderThan = 60m

I can see stuff with sourcetype=glassfish:access for the Last 15 minutes but nothing since 03.12.2016 for glassfish:gc

I have checked that the path is correct by checking the global zone(solaris) that file exists:

/root/opt/SUNWappserver/glassfish/domains/scdomain/logs.

I have also done telnet tests for all the indexers on port 9997 and works and the deployment server on 8089 works also.

0 Karma

woodcock
Esteemed Legend

You can use ingnoreOlderThan but if you do, beware that it does not work the way most people think that it does: once Splunk ignores any file for this reason at any time, it is put into a permanent blacklist and it will never be examined again, even if new data goes into it! This is likely your problem so just remove ingnoreOlderThan. If the files still don't go, then you may need to change their names to get past the previous effects of the blacklisting.

0 Karma

maxruas
New Member

Hi
thank you for you kind response, Is there a way that I can check what is been permanently black listed.
If so can I remove entries from that list?

thanks

0 Karma

woodcock
Esteemed Legend

I have no idea.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...