Getting Data In

How to forward specific log files to specific indexes?

dwoehr
Explorer

Hello, I'm trying to figure out the following setup:

At the moment we have one rotating log file that should be forwarded to one specific Splunk index / source type. But this will most likely not stay the only log we want to have in Splunk, but the other logs we will be forwarding later, should end up in another index / source type.

From my understanding, I would have to set up a monitor in the inputs.conf for each file or folder I'd like to monitor. What I don't get is, how to set the target index/source type for that monitor in the outputs.conf, or if I'm on the right way at all with my assumption.

So basically, what I'm trying to accomplish is to set up the Universal Forwarder to do the following:
Forward:
our_first.log to index 1 / source type 1
our_second.log to index 2 / source type 2
and so on

Is there a finished example anywhere how to get this done? I can't figure out the connection of the inputs.conf and outputs.conf from the documentation.

Thanks a lot

0 Karma
1 Solution

koshyk
Super Champion

In your inputs.conf, (eg let's say the files are in /var/log/myapp/)
Both sourcetype and index is set at inputs.conf

[monitor:///var/log/myapp/our_first.log]
sourcetype=sourcetype1
index = myindex1

[monitor:///var/log/myapp/our_second.log]
sourcetype=sourcetype2
index = myindex2

View solution in original post

koshyk
Super Champion

In your inputs.conf, (eg let's say the files are in /var/log/myapp/)
Both sourcetype and index is set at inputs.conf

[monitor:///var/log/myapp/our_first.log]
sourcetype=sourcetype1
index = myindex1

[monitor:///var/log/myapp/our_second.log]
sourcetype=sourcetype2
index = myindex2

dwoehr
Explorer

Thanks,

I configured one monitor like the example now and have an outputs.conf like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = xxx.xxx.xxx.xxx:9997

[tcpout-server://xxx.xxx.xxx.xxx:9997]

With:

list forward-server

I can see the following result:

Active forwards:
        xxx.xxx.xxx.xxx:9997
Configured but inactive forwards:
        None

And with:

list monitor

I can see my monitored file

But there's no new events in Splunk. I have port 9997 active in Splunk under settings -> Forwarding and receiving -> Receive data

Any ideas where to start searching for errors? Is there an error log I could check? Or any possibility to see if the universal forwarder is even trying to forward events?

Thanks again

0 Karma

koshyk
Super Champion

you can check lots of places. Do you have iptables blocking port 9997?
- check _internal index on your master Splunk to see if handshake is made . Handshake normally happens on management port 8089
- login to the Universal forwarder and check if any errors are present in "splunkd.log"
- Try logging to the UF and do a ssh -v -p 9997 {ip_of_indexer} to see if it contact the 9997 port
- More advanced issues like ulimits etc (but these won't happen unless its a huge prod system with lot of files open in tandem)

0 Karma

dwoehr
Explorer

OK, thanks, I'll keep that in mind for future problems. But I just logged in this morning and the events were there. I suppose they were initially not fast enough to show up. But now we get the events in near realtime. Which is what we were looking for.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...