Tagging data from different sources that will send to same forwarder...

[rjlaral]

Hello All:

Greetings. First of all, thanks for take your time and read this question; apologies by my poor English.

I'm really N00b about Splunk Enterprise and Splunk Forwarder, but I must be involved in a project in my work, so I must be learn!!!. I deeply read this article (https://answers.splunk.com/answers/333248/how-to-implement-tagging-on-a-universal-forwarder.html), this (https://answers.splunk.com/answers/129225/tag-data-on-universal-forwarder.html) and this (https://answers.splunk.com/answers/501121/how-to-add-custom-tags-to-event-data-via-universal.html) related to my question, but I don't understand very well those answers.

This is my scenario: In premise, a server is installed with a number of virtual machines (VMs), one of them have Splunk Forwarder installed. Other VMs have applications generating logs we want to analyze. In other site, we have another server running Splunk Enterprise.

We want to collect information from VMs and send to Splunk Enterprise via Splunk Forwarder for futher analysis, but we want to tag this information to categorize it and do analysis. For example, if VM1 run App1 and generating LogApp1, we want to send this data to Splunk Enterprise and be able to do several search, analysis, filters over LogApp1 data.

So:

1. What is the right configuration over log data in VMs and Splunk Forwarder to tagging, categorize it in order to identifying in Splunk Enterprise? I'm thinking in use syslog to send data from VMs o Splunk Forwarder, but How I must tag this data in order to get a tagged data and not a mess?
2. Can you provide me full example in how I must configure VM, syslog (if this is part of solution), Splunk Forwarder and Splunk Enterprise (Is indexers involved here?)

Please, I really N00b about Splunk Architecture and any help will be appreciated, even if you consider basic your support, guide, advice or answer. I really open to Splunk 101 answers.

Please, let me note if you need further explanation about my problem, I hope I had described enough.

Thank you very much and best regards,

Rafael

[rjlaral]

Ciao Signor Cusello, Greentings Mr. Woodcock:

First of all, thank you for your responses! and my apologies for my later response.

It's my fault, but I also need to configure other (real hardware) servers to send logs to Splunk Forwarder's VM so it can send all collected logs to indexer. And, I face same issue: categorize this logs for further analysis and identify app-log relationship. So, in scenario, Splunk FW VM not only collect logs from other VMs (and you're right, Mr. Woodcock, is VMWare ESXi) but servers outside the VMWare enviroment.

My real problem is I don't understand how Splunk architecture works and, by the way, including syslog functionality; but the last one is more easiest than Splunk. I consider Splunk documentation is the hard way for learn Splunk from scratch. Signor Cusello, thank you for URLs you indicated in your response.

Can you provide me several specific points to go in the right direction, guys? Maybe further advices? Examples? I really appreciate that. I'll keep an eye on your answers.

Thank you very much and best regards,

Rafael

[woodcock]

I am assuming that you are trying to send information about the VM that is being collected by VMWare, right (otherwise, just install a Splunk UF on each VM and sent it directly). If so, you need to look at Splunk app for VMWare on splunkbase.

[woodcock]

Now that I know that we are talking about VMWare, then I have bad news. Getting that stuff setup and working is one of the more complicated adventures in Splunk for a new user if you are trying to get information from vCenter. If you just need ESXi, then here is a series of blog posts that will walk you through it:

https://www.virtualtothecore.com/en/vmware-admin-splunk-noob-install-and-configure-splunk/

[gcusello]

Hi rjlaral,
the easiest way to proceed is to install a forwarder on each VM and configure them to send logs to you indexer.
In this way you have a very easy forwarders management, you can recognize VMs (VM's hostname) anche you have all the features of a forwarder (cache, easy monitoring, easy configurations, etc...)
if you don't want to install a faorwarder in every VM, you have to configure each VM to send logs to the indexer (you don't need a forwarder) by syslogs.
but remeber that syslogs aren't repetable so, if there is a network problem or Splunk server is in maintenance you lose logs.

For more information see https://docs.splunk.com/Documentation/Splunk/latest/Data/Usingforwardingagents

If your VMs are less than 50, I suggest to you to use your Splunk server also as Deployment Server so you can deploy configurations to alla VMs from a single point.

For more information see http://docs.splunk.com/Documentation/Splunk/6.6.2/Updating/Aboutdeploymentserver

Bye.
Giuseppe