Getting Data In

NullQueue and Re-indexing Logs

asarolkar
Builder

I want to limit the data from one of my universal forwarders from being indexed temporarily (until we get a new Splunk Daily Limit override).

Note that I do not want to block 9997 on the indexer. I do not want to shut down splunk on the forwarder either.

The remedy which involves writing all logs to nullQueues is a potential solution:

http://splunk-base.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue

My understanding is that when we write logs from a certain sourcetype (or in this case ALL sourcetypes) to a nullQueue, it will mark it for indexing later.

The key here is that I want it to go to a nullQueue NOW, but when then throw-away the nullqueue configuration and re-index everything that was sent to the nullQueue.

Is that how it works ?

Any other suggestions if going about it in this manner is not useful ?

Thanks

0 Karma
1 Solution

Ayn
Legend

No, that is not how the nullQueue works.

nullQueue works like /dev/null on *NIX systems - data sent there is gone, never to be seen again. Splunk does not keep track of what's been sent there either. Without shutting down your forwarders or blocking them from accessing the indexer, I don't see an obvious solution to your problem.

View solution in original post

Ayn
Legend

No, that is not how the nullQueue works.

nullQueue works like /dev/null on *NIX systems - data sent there is gone, never to be seen again. Splunk does not keep track of what's been sent there either. Without shutting down your forwarders or blocking them from accessing the indexer, I don't see an obvious solution to your problem.

asarolkar
Builder

Forgive me for asking but I am not entirely sure where to set that on the forwarder config.

I am guessing its in one of the conf under /etc/system/local.

0 Karma

Ayn
Legend

That should work. Just make sure you have proper settings for how events are buffered on the forwarders, so you're not losing events if the buffers fill up.

asarolkar
Builder

Hi there:

We are assessing the efficacy of a third-way solution.

This involves putting a network firewall between the forwarder and the indexer. Data will keep getting pushed to the indexer on 9997 but will never make it all the way to the indexer.

This way that data is not "lost" and will be re-indexed when this temporary firewall is lifted AND we are not blocking 9997 for all the other forwarders that want to push data.

What say you ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...