Getting Data In

Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99
Explorer

Hi,

I am new to Splunk and needs to take care of existing Splunk setup. I am trying to forward large CSV file from Universal Forwarder(UF) to Heavy Forwarder(HF) and then indexer.
Our existing setup is 1 MA , 2 HF, 2 SH and 2 INX. right now I am forwarding live logs from Arcsight server (non splunk forwarder) to HF which divide into 10 different indexes based on data type.

Without disturbing the above setup I want to forward large CSV files (5-10MB) from Linux server using UF.

See below the config settings that I did so far, I am receiving events on HF but not sure how to redirect them to a specific index.

on Universal forwarder

**input.conf**

[default]
host = server1.mydomain.com
[monitor://opt/client/reports/archive/Splunk/]
sourcetype = csv
index = main

**Output.conf**

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = hfserver2.mydomain.com:6514
[tcpout-server://hfserver2.mydomain.com:6514]

On Heavy Forwarder

**input.conf**

[default]
host = hfserver2.mydomain.com
# receive events from the ArcSight Forwarding connector
[tcp:4514]
disabled = 0
sourcetype = cefevents

#For Reports (to receive events from Splunk universal forwarder ) 
[splunktcp:6514]                                                                 
disabled = 0
sourcetype = csv


**Output.conf** (I haven't made any changes here ) 

# use indexer discovery to identify the Indexers dynamically
[indexer_discovery:splunk_master]
pass4SymmKey = ********************************
master_uri = https://ma1server.mydomain.com:8089

# don't index any events on the Heavy Forwarder, just forward events to the Indexers
[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers

[tcpout:SplunkIndexers]
# identify which Indexers to send events to by querying the master node
indexerDiscovery = splunk_master

With this setup, I am receiving events on HF (verified via tcpdump) but I can't see it on the indexer. I think its getting discarded.
I am doing all the changes in config files since I don't know how to do it on Master's web UI.

Please help me.
Thanks in advance

0 Karma
1 Solution

sanjayjp99
Explorer

I am searching data on search head and indexer, I have also tried capturing packet in HF and indexer.
No, master is not indexing data.

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false

View solution in original post

0 Karma

sanjayjp99
Explorer

I am searching data on search head and indexer, I have also tried capturing packet in HF and indexer.
No, master is not indexing data.

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false
0 Karma

sanjayjp99
Explorer

I am searching data on Search head and indexer.
No, master is not indexing data.
In above config I am using master's url for indexer discovery and as I said its working for other indexes. (port 4514)

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false
0 Karma

abhishekkoli
New Member

HI sanjayjp99,

As per given config files data is moving like UF-->HF-->MA.

You need to make conf file in order to send data from UF-->HF-->INDEXER.

Make sure you are sending data to indexers from the cluster.

0 Karma

abhishekkoli
New Member

HI
Where you are searching for the data ? (MA or SH)
IS MA is used to index data ?
WHat is output cong for MA ?

0 Karma

sanjayjp99
Explorer

As per given config files data is moving like UF-->HF-->MA

I think that's how you send data in cluster environment, sending data straight to indexer will be limited to that specific indexer will not be cluster. correct me if I am wrong.
above setting is working for indexes that are coming to HF via non Splunk forwarder.

Thanks

0 Karma

abhishekkoli
New Member

Hi sanjayjp99,
Data should point out all the indexers in the cluster not Indexer master.
Indexer Master will decide the which indexer has to receive data .

  1. UF-->HF-->MA this is not recommendation.
  2. UF-->HF-->IDX1,IDX2- Point the data to all indexers in cluster through output file of heavy Forward.(IDM) will take care of Replication and search factor.
0 Karma

sanjayjp99
Explorer

If I send direct to my indexer it will not be cluster right?

0 Karma

sanjayjp99
Explorer

still waiting on possible solution

0 Karma

woodcock
Esteemed Legend

You should not be indexing this way unless you have a reason to. You should be sending directly from your UF to your Indexer tier. Why are you not doing this?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...