Quarantined heavy Forwarder

[gcusello]

Hi at all,

I encountered a strange behaviour in one Splunk infrastructure.

We have two heavy Forwarders that concentrate on-premise logs and send them to Splunk Cloud.

Form some days, one of them stopped to forwarder logs, also restarting Splunk.

I found on both the HFs three new unknown folders: quarantined files, cmake, swidtag.

In addition, sometimes also the other HF stops to forward logs and I have to restart it and the UFs, otherwise log collecting stopped.

I knew thet an Indexer can be quarantined, also an Heavy Forwarder?

How to unquarantine it?

I opened a case to Splunk support, but in the meantime, Is there anyone that experienced a similar behavior?

Thank you for your help.

Ciao.

Giuseppe

[JohnEGones]

Hi guys,

Great discussion, it is both interesting and insightful to get to see and "listen in on" experts having both problems and being willing to do so publicly. Thank you.

Cheers,

[PickleRick]

That's... strange.

As you know (and @isoutamo  already pointed out as well), you quarantine search peers on your search head(s) so that the searches do not get distributed to that search peer. So HF shouldn't have anything to do with quarantine.

swidtag directory is a part of normal Splunk distribution and has been around for a long time. If you didn't have it before... Are you sure someone didn't try to ineptly "upgrade" your Splunk installation?

[gcusello]

Hi @PickleRick ,

I'll try to upgrade to 9.1.3, hoping that it will solve the issue!

Ciao.

Giuseppe

[PickleRick]

It was not my suggestion. I was asking whether someone didn't try to upgrade or do something else with that installation so that it was modified unbeknownst to you.

[gcusello]

Hi @PickleRick ,

for my knowledge, there wasn't any upgrade,

in few minutes I'll have a call with Splunk Suppot: I hope well!

Ciao.

Giuseppe

[isoutamo]

Hi

Quite interesting behaviour. As HF is basically an indexer without local indexing I don't see any reason why it cannot quarantine? But interesting part is who has set it as quarantine as usually this is done by search peer. And as quarantine actually means that this search peer shouldn't part a searches it shouldn't affect any indexing/forwarding function. One normal way to to use quarantine is just ensuring that peer can index/transfer full queues without disturbing by searches.

You probably have already read this https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Quarantineasearchpeer

Have you local MC or just CMC in use? If 1st one, have you check if MC has marked is as quarantine?

r. Ismo

[gcusello]

Hi @isoutamo,

thank for your help!

Yes I already saw the above link, for this reason I opened the case: because in the url is described an action on the search head, but I don't have SHs and in HFs distsearch.conf there isn't the described lines.

I suppose that's a quarantine issue because I have many messages in splunkd.log that speaks of quarantined files, but I don't know how to unquarantine the machine.

I'm waiting for the call from Splunk Support, hoping that they can guide me.

Have you never exeperienced this issue?

Local MC doesn't give any quarantine message, only that "the downstream queue is not accepting data", but I can reach Splunk Cloud by telnet, so it isn't a firewall issue.

Thank you again, please hint every check that you can think (if you have).

Ciao.

Giuseppe

[isoutamo]

I haven't seen this before.

But your keywords this is what pop up from google https://github.com/wazuh/wazuh/issues/21383