Splunk Search

Data forwarded to third party system missing metadata

calinm
Engager

Hi,

I have an all in one enterprise splunk install (indexer, search head, file monitoring) with a number of universal forwarders forwarding data to it.

I need to forward everything I receive on the central splunk server to a third party system. I tried both the:

[syslog] (using TCP)
and the
[tcpout]

approaches described in the doc.

The log data is forwarding fine.

The problem is that the metadata is missing. The source,sourcetype and host are missing when using tcpout and the source and sourcetype are missing when using syslog. Without them, the data will make no sense to the receiving system.

Is it possible to use a transform to put the metadata back in the data? If so, How?

Thanks!

Calin

0 Karma

Jeremiah
Motivator

Yes its missing unfortunately.

http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Forwarddatatothird-partysystemsd

If you need to move data to another system with included metadata you might want to have a look at the dump command or the Hadoop Connect app. You can export to a mounted filesystem if you don't actually have an HDFS cluster running:

http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/Dump
http://www.splunk.com/en_us/solutions/solution-areas/big-data/splunk-hadoop-connect.html

0 Karma

calinm
Engager

That's interesting, Thanks Jeremiah. I did not know about the dump command, but I don't think it helps here because I need to stream data, not move data.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...