All Apps and Add-ons

UDP logs going to messages file but not indexing -- why?

pfabrizi
Path Finder

I have a Cisco ASA sending to my universal forwarder. However I noticed that as of May the events stopped indexing.

Looking at my /var/log/messages file I noticed that all the messages are going there, how do I correct this?

Cisco is send UDP.

0 Karma
1 Solution

tmarlette
Motivator

welp.... if you have a network appliance sending syslog on port 514 to a universal forwarder, there may an issue right there. While a UF can listen for data, it can't do anything to redirect it to indexers cooked properly.

I'd check a couple of things.

  1. is rsyslog on this machine running, and receiving this data as well and dumping it into /var/log/messages?
  2. install a HF on this machine instead, and then use inputs/props/transforms to redirect all data from that specific IP address to a different index / sourcetype.

View solution in original post

0 Karma

tmarlette
Motivator

welp.... if you have a network appliance sending syslog on port 514 to a universal forwarder, there may an issue right there. While a UF can listen for data, it can't do anything to redirect it to indexers cooked properly.

I'd check a couple of things.

  1. is rsyslog on this machine running, and receiving this data as well and dumping it into /var/log/messages?
  2. install a HF on this machine instead, and then use inputs/props/transforms to redirect all data from that specific IP address to a different index / sourcetype.
0 Karma

pfabrizi
Path Finder

so because I am using a UF I can't forward anything? What if I can get the CISCO UDP port changed?

can I send directly the 1 of my 2 indexes instead?

Thanks!

0 Karma

pfabrizi
Path Finder

I resolved this issue by adding an entry to the rsyslogd to forward any events from my specified IP (ASA appliance) to a cisco log. I knew this would work as we forward other UDP (514) events to SPLUNK this way.

0 Karma

tmarlette
Motivator

You can forward local logs to your indexer from a UF, but if you want to listen on other ports, you'll need a heavy forwarder. It's always good to have a heavy forwarder in an environment for these types of things, or any other modular data inputs that you may need to use, like AWS or something. That way, you can assure the incoming data stream is following your standardized outputs.conf for you whole environment.

If you want, you can open up the UDP port on one of your indexers, or even on a search head, yes, however direct to the indexer is generally not used.

If you have a cluster, then all of your data is in one spot, and it lacks any cluster mappings for it's buckets.

if you added an outputs.conf to your search head, and opened the listening port there, you would be able to do it, though the resources of the box would take a slight hit for doing the extra data forwarding.

0 Karma

pfabrizi
Path Finder

so if those ASA logs are posting in the messages log file on my universal forwarder can those now be considered local logs and be forwarded ?

if so would it be a configured as syslog?

Thanks!

0 Karma

tmarlette
Motivator

Yes they can. It would be a standard [monitor:///var/log/messages] stanza.

You'd have the seperate the data at the indexers using data routing / filtering techniques if you wanted standard host level /var/log/messages data on a different index.

http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...