Getting Data In

How to index only a lower directory on a path shared by another index?

Bliide
Path Finder

I have a couple of indexes that are pulling the same data. One index pulls local data and one is pulling data from a server using a universal forwarder. The data resides on different systems but the path on each system is similar. The Index I have setup for the forwarder data is pulling in the correct data but it is also pulling in data locally that shares the same path as the forwarder. For example:

I have 2 indexes, one is getting data locally and the other from a universal forwarder. IndexA is local and IndexB is pulling from the forwarder.

IndexA has log files that are in the following example local path: D:\Log Files\App Name\Web Logs

IndexB has log files that are in the following example path on a server with a universal forwarder: D:\Log Files\App Name

In inputs.conf on the indexer and on the server using a universal forwarder I have a monitor with the following setup:

[monitor://D:\Log Files\App Name]
disabled = false
index = IndexB
sourcetype = IndexB
Host = Server-Name

IndexA has the correct data from the local directory D:\Log FIles\App Name\Web Logs. IndexB has the correct data from D:\Log Files\App Name but it is also pulling in data on the local path D:\Log Files\App Name\Web Logs.

What can I do to get the forwarded data to be the only data pulled into IndexB? I know I am missing something that is probably obvious but I can not see the forest for the trees. Any help is appreciated.

I have

0 Karma
1 Solution

chanfoli
Builder

Data is not pulled by the indexer. The inputs.conf on your forwarder tells it what data to send to the indexer. Your forwarder inputs need to only be defined on your forwarder. It will push data as defined by your monitor stanza to your indexer(s) as defined by outputs.conf. It seems to me that this is where your problem lies. The indexer does not need duplicated input stanzas, the inputs on the indexer apply to the local system and monitor stanzas will be treated the same, meaning that you need to set up different monitors on each to get the files indexed as you desire.

View solution in original post

chanfoli
Builder

Data is not pulled by the indexer. The inputs.conf on your forwarder tells it what data to send to the indexer. Your forwarder inputs need to only be defined on your forwarder. It will push data as defined by your monitor stanza to your indexer(s) as defined by outputs.conf. It seems to me that this is where your problem lies. The indexer does not need duplicated input stanzas, the inputs on the indexer apply to the local system and monitor stanzas will be treated the same, meaning that you need to set up different monitors on each to get the files indexed as you desire.

Bliide
Path Finder

It worked. I created a new index name in an edited monitor stanza. IndexB is now pulling in the correct data but it is pulling data that was not indexed by the old setup. How can I tell IndexB to index all data in the log files that was already indexed previously?

0 Karma

chanfoli
Builder

Hello, I edited your question to show the backslashes. Why is the higher level monitor stanza also defined on your indexer? It would seem that if you wanted all files under "App Name" on the forwarder to get indexed into IndexA but only want files under "Web Logs" on the indexer to get indexed in IndexB, you should have a different upper level monitor stanza defined on your forwarder and a more specific stanza looking at "Web Logs" on your indexer, pointing to Index A.

Does that make sense?

0 Karma

Bliide
Path Finder

The app on the forwarded server creates a log file in the "App Name" directory on that server. That is the most distinct path I can give the Universal Forwarder for the logs. On the index machine I have logs that follow that same path but have a more distinct directory structure after "App Name".

I thought I had to put a monitor in inputs.conf on the indexer so that it would pull in the data from the universal forwarder. If I do not need a monitor in inputs.conf on the indexer for the forwarded data that may be problem.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...