Getting Data In

getting access to raw events on forwarders

ViewPoint
Engager

Our application requires access to raw events on light forwarders to do some custome processing before (or at the same time) as the events get passed to central indexer. Is there any way to tee the even stream on forwarder to split the stream into two destinations - the splunk indexer AND our separate processor? If so, is it possible for us to get some kind of the ID that is (will be) assigned to the current event so our database can have a reference to the original even as it is being added to the database?

MikhailArefiev
Explorer

It is possible to forward raw data to third party servers by configuring $SPLUNK_FORWARDER_HOME/etc/system/local/outputs.conf on the forwarder instance in this manner:

[tcpout]

[tcpout:fastlane]
server = 10.1.1.35:6996
sendCookedData = false

The last line prevents the data from being prepended with additional timestamps.

Source: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwarddatatothird-partysystemsd

Note: $SPLUNK_FORWARDER_HOME is /opt/splunkforwarder for a typical Linux installation.

yannK
Splunk Employee
Splunk Employee

The lightweight and universal forwarder do not parse the events.
If you want to use splunk to process the events, use an heavy forwarder with props and transforms.

Beware the data will be parsed when send to the indexer who will not parse then twice, so make sure that your heavy forwarder as all the rules that you also apply on the indexers.

see http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Deployaforwarder

0 Karma

yannK
Splunk Employee
Splunk Employee

About your double processing (in splunk and in your database)
You can send a copy of some events to a third party system as raw or syslog.
see http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Forwarddatatothird-partysystemsd

For the unique ID to identify the same events in splunk kand in the third party system, this requires that you added the unique id yourself to your events. (probably in the original log before the indexing)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...