Splunk Search

How can I search on _internal logs from forwarders in my environment?

yonphang
Explorer

Hello everyone,

It seems like I couldn't find any previous answer on this from the community. I have more than 1000 forwarders installed in Windows/Unix servers. I do not have any RDP nor SSH access into those servers due to security reasons. Once in a while, forwarders do not ping back to the server, so I need to access the Splunk "logs" in the directory without having to RDP nor ssh into those servers. I was told that there's already an index which does the internal logging thing and it's doing so in all forwarder agents.

So I was trying to run an index=_internal search in the deployment server, but it returned me with just a hostname, and the hostname was the hostname of the deployment server. Am I doing it wrong? I was trying to check the _internal logs from the forwarders.

Can anyone shed me some light on this?

Thank you

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi yonphang,

in the docs about forwarding search head internal logs http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata you can find the instruction to do so.

The important part is to add forwardedindex.filter.disable = true in outputs.conf of each of the forwarders and restart them. After that you will be able to search them in index=_internal

Hope this helps ...

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi yonphang,

in the docs about forwarding search head internal logs http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata you can find the instruction to do so.

The important part is to add forwardedindex.filter.disable = true in outputs.conf of each of the forwarders and restart them. After that you will be able to search them in index=_internal

Hope this helps ...

cheers, MuS

yonphang
Explorer

does this means the forwarder agent will forward it's splunk log to the master server?
I don't know if the local log gets forwarded automatically?
can you provide me more detail on this?

Thanks a lot

0 Karma

MuS
SplunkTrust
SplunkTrust

Yes, by setting this option the internal logs of the forwarder will be forwarded to the server set in the outputs.conf of the forwarder.
More details? Hmm, the basics are the same if you forward search heads internal logs or forwarder internal logs - so I don't know if there is more detailed information available....sorry

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...