Getting Data In

Configure a splunk searchhead so i can query the forwarder

dominiquevocat
Motivator

I have a deployment server from where i have a firewall rule that alows me to reach the 8089 management port of all forwarders. I would like to use the |rest server=xxx command to list porperties of the forwarder.
From command line i can successfully query al rest endpoints of the remote splunk so that part is ok.
When i use the |rest command i get a information "Search filters specified using splunk_server/splunk_server_group do not match any search peer."
I figured i might have to (gasp) configure all remote hosts as distributed search heads? But i encounter an issue doing so:
"Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer: Connection reset by peer".
What am i fundamentally doing wrong? I don't need to write a custom command to query the splunk endpoints do i? Or how do you guys inspect your forwarders???

1 Solution

dwaddle
SplunkTrust
SplunkTrust

I don't think the forwarder license allows it to participate as a distributed search peer. That (could) be the source of your reset error - even though that would be a weird error to get. And even if the forwarder does allow it, I've never heard of anyone trying to do anything like this. "Number of search peers" is a variable in search response time ; adding a bunch of forwarders as search peers could make this really bad. There are tunables that could help here (multi-threaded setup?) but this is probably a losing strategy in the long run.

Most admins I've worked / talked with do not "inspect" forwarders. It's not uncommon to disable the REST port on forwarders entirely, or at least limit it to the loopback interface. Then, if you need it for troubleshooting purposes then you enable it on purpose.

What exactly are you trying to accomplish here?

View solution in original post

0 Karma

dominiquevocat
Motivator

I made an app to do this: https://splunkbase.splunk.com/app/2775/

0 Karma

dwaddle
SplunkTrust
SplunkTrust

I don't think the forwarder license allows it to participate as a distributed search peer. That (could) be the source of your reset error - even though that would be a weird error to get. And even if the forwarder does allow it, I've never heard of anyone trying to do anything like this. "Number of search peers" is a variable in search response time ; adding a bunch of forwarders as search peers could make this really bad. There are tunables that could help here (multi-threaded setup?) but this is probably a losing strategy in the long run.

Most admins I've worked / talked with do not "inspect" forwarders. It's not uncommon to disable the REST port on forwarders entirely, or at least limit it to the loopback interface. Then, if you need it for troubleshooting purposes then you enable it on purpose.

What exactly are you trying to accomplish here?

0 Karma

dominiquevocat
Motivator

Um, remote query monitored files and all sorts of settings that a (linux-)Admin might have "tweeked" on the server and thus causing me issues. The rel issue why i started to look into it was weird host names which i found out to be the package name in the zypper.log

Still, it would be nice to be able to use the rest command to query all sorts of properties of a splunk using the standart command, no? The xml from the rest interface is suffiently complex so that i would not want to reimplement it...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...