All Apps and Add-ons

How can I index logs from ESXi hosts? And how do I install the Splunk Add-on for VMware?

arindrew
Explorer

First off, I've been using Splunk for about two weeks so I am not all that familiar with how things are suppose to work.

I have Splunk Light installed on Linux (and so far, it is receiving forwarded logs from two windows servers and two linux servers as well as syslogs from our two firewalls. So far, so good.

I am trying to get Splunk to receive logs from our three ESXi hosts (two are 5.5 and one is 6.0) without any luck. I tried forwarding them to the default 514 port on Splunk, but Splunk said it was unable to create a listener on that port (due to not running as root from what I've read), so I configured ESXi to send its syslog to the same listener (on port 33514) that I am using for the firewall syslogs, but it doesn't look like I can modify the firewall on ESXi to send to that port (only 514 and 1514 are options).

So I created a second listener on port 1514 on Splunk, which worked, but when I configured ESXi to send to that host:port combination, Splunk doesn't receive the data.

I read around, and found that there is a Splunk Add-on for VMware - but couldn't find it in my Add-on lists. I'm confused about how to install this, or where.

I also found an OVA for a Splunk DCN and installed that on one of the ESXi hosts. Went through the configuration and got this error:

License master configuration: Fail
In handler 'localslave' : editTracker failed, reason='WARN: path=/masterlm/usage: This license does not support being a remote master. from ip=10.25.1.24'

So now I am just really lost on which method is the recommended method to get ESXi syslogs to a Splunk Light server.

0 Karma

ww9rivers
Communicator

Splunk Add-on for VMware is a collection of TAs (add-ons) that Splunk uses to collect, parse and transform data for other Splunk Apps.

This page shows an "architectural diagram" of how various components are supposed to work -- hope that helps.

0 Karma

arindrew
Explorer

Looking at that diagram, I have Splunk acting as a "Syslog Server" already. Can I not just set ESXi to send its logs to that same server?

0 Karma

ww9rivers
Communicator

You should be able to -- although I have not used "Splunk Light" before.

Have you gone into Settings >> Data Inputs to see if you have added and/or enabled TCP/1514 and UDP/514 for syslog reception?

Other than that, I would try to see if you do receive data on those ports on the Splunk host from your ESXi host with something like "tcpdump".

Also, Splunk's logs in /opt/splunk/var/log/splunk/splunkd.log may also help to troubleshoot.

One more possibility is that syslog may not have the correct time stamps so you may also try to search with "All time" window on Splunk, if you are sure that data should be received in Splunk.

0 Karma

arindrew
Explorer

I did try to install the TAs into my Splunk server, but got as far as extracting the gzip into /opt/splunk/etc/apps/ and not knowing what to do from there.

I went into my Splunk Data Inputs to try and add a TCP input on 1514 since I was originally planning on using UDP.
I see that two inputs are already created (possibly by the OVA?):

1514 with a source type of vmw-syslog
1517 with a source type of vclog

Both are disabled, and when I enable the 1514 input, I receive the following error:

Error occurred attempting to enable 1514: In handler 'raw': Could not find writer for: /nobody/Splunk_TA_esxilogs/inputs/tcp://1514 [0] [/opt/splunk/etc].

0 Karma

ww9rivers
Communicator

I have been using rsyslog as syslog server, so I don't know exactly how the Splunk_TA_esxilogs add-on is configured -- Try to follow this and see if you could get it going.

My guess is that, Splunk_TA_esxilogs takes data in. It then will need to either write the data to a file or an internal pipeline to another Splunk module.

Your Splunk Light server may be configured out-of-the-box to forward data to itself for indexing. You may run "splunk btool outputs list" and search for "defaultGroup" to see where your data goes.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...