Getting Data In

Indexer receives forwarded Splunk internal logs but does not receive non-Splunk os events

Ant1D
Motivator

Hi,

I set up a Linux forwarder to forward os logs to a Windows indexer as a test. The Windows indexer is running Splunk Free (500MB per day license limit).

After setting this up I am only seeing Splunk internal logs (e.g. _internal) when querying data on this indexer. The index (index=myindex) on the indexer which I have configured to receive the os logs is empty. Nothing has been written to this index. The forwarder has been configured to forward data to index=myindex.

Any ideas why my indexer is indexing internal logs from this forwarder but is not indexing os logs?

This forwarder is also forwarding the os logs to a separate Linux Splunk instance and it is working as expected there (i.e. This Linux Splunk instance indexes both os and Splunk internal logs).

Thanks

Tags (2)
0 Karma
1 Solution

Ant1D
Motivator

The issue was with outputs.conf.

There were a number of outputs.conf configuration with similar tcpout stanzas which were causing some config to take precedence over other configuration.

View solution in original post

0 Karma

Ant1D
Motivator

The issue was with outputs.conf.

There were a number of outputs.conf configuration with similar tcpout stanzas which were causing some config to take precedence over other configuration.

0 Karma

woodcock
Esteemed Legend

Either:
1: You did not setup your inputs.conf correctly on your forwarder. If so, there should be errors in _internal from your forwarder that indicate this.
2: You did not setup myindex correctly on your indexer. If so, there should be errors in _internal from your indexer that indicate this.

0 Karma

Ant1D
Motivator

Both of these were setup correctly. I have fixed the issue.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Ant1D,
Receiving _internal logs means that you correctly configured connection between UFs and Indexer, but probably there's an input error: what did you used on Universal Forwarder for input: TA_nix or a manual inputs.conf?
Then which index did you configured on inputs.conf in UFs?
If you used TA_nix, by default it sends unix logs to os index.

Ciao.
Giuseppe

0 Karma

Ant1D
Motivator

The index=index_names in inputs.conf on the Linux forwarder match the indexes created on the Windows indexer where the data is not being indexed. Yet this Windows indexer is having no issue indexing _internal logs from the Linux forwarder.

It is a manual inputs.conf on the forwarder. Both the _internal and os logs get indexed fine in the separate Linux Splunk instance but the Windows indexer only chooses to index the _internal logs. Both the Windows indexer and the separate Linux Splunk instance have the same indexes created.

This is weird

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Ant1D,
could you share your inputs.conf?
Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...