Getting Data In

How to stop indexing forwarded data from heavy forwarder that indexes locally

damindragunatil
Explorer

Reading from article : Does data indexed and forwarded from a heavy forwarder to indexer would charge twice?

Any indexed forwarded events from a Heavy forwarded are NOT licensed twice.

When Indexing and forwarding from a Heavy Forwarder, the licensing is only used at the Heavy Forwarder, since indexed Data sent to the Indexer, doesn't go through the Parsing queue (as well as the Aggregator and Typing queues).

I have setup the following on my Heavy Forwarder:

outputs.conf:

defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = rdbrsdem03.ref.clp7.local:9997
indexAndForward=true

props.conf

[source::tcp:9999]
BREAK_ONLY_BEFORE=^CEF\:0\|

So on my heavy forwarder, I am sending indexed data to my indexer (rdbrsdem03), and it also filters all events that start with CEF:0|

When I check licensing it seems as if the events ARE being indexed on both the Heavy Forwarder and Indexer.

Can someone provide me with a search possibly using the 'summary' index that proves the events are only being index at the Heavy Forwarder, please?

I have a developer license at the moment so would like to prove that events that need to be indexed at the Heavy Forwarder (due to local users in a remote site being able to search events of their local hardware events) and then not being reindexed (in effect doubling licensing costs) on the Indexer.

Hope this all makes sense, please let me know if there is anything further you may need.

kind regards

Damindra

0 Karma

damindragunatil
Explorer

| NODE| IDX? | FWD? |
2. +-----+--------+--------+
3. | HF | YES/| YES|
4. | IDX | YES/ | N/A |

Hope this makes sense, the reason is there needs to be local searching on the HF.

What would you advise in regards to the LINE_BREAKING?

thanks

0 Karma

woodcock
Esteemed Legend

You have no configurations that "filter". The BREAK_ONLY_BEFORE=^CEF\:0\| is a (poorly-performing) LINE_BREAKING configuration. Even so, I am unclear on your goal. Please fill out this chart:

| NODE|  IDX?  |  FWD?  |
+-----+--------+--------+
|  HF | YES/NO | YES/NO |
| IDX | YES/NO |   N/A  |
0 Karma

skalliger
SplunkTrust
SplunkTrust

We recently had this discussion on the Slack usergroups. A heavy forwarder doing indexing is an *indexer. * License usage gets applied when events get written to disk. This means, when you index twice, your license gets hit twice also.

Skalli

0 Karma

damindragunatil
Explorer

Hiya, the source of the answer was here on Splunk Answers

https://answers.splunk.com/answers/337523/does-data-indexed-and-forwarded-from-a-heavy-forwa.html

kind regards

Damindra

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Thanks for the citation. That answer has since changed.

Information on Answers is not official and not always definitive. See this answer: https://answers.splunk.com/answers/506909/heavy-forwarder-as-indexer-and-license-usage.html
I'm struggling to find this mentioned in official Splunk docs.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where did you read that index-and-forward does not count twice against your license? I believe that's incorrect, but would like to see your source.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...