Lookups in SA-VMW-HierarchyInventory

[stepmmx]

Hi,
to build the TimeClusterServicesAvailability lookup you use the following query:

index=vmware source="ClusterComputeResourcePerf" perftype=clSvcs.....

but I don't have any events with source="ClusterComputeResourcePerf"

Which perl script generates this events?

[stepmmx]

I basically asked about the same subject already (with little debugging information)

http://splunk-base.splunk.com/answers/72666/vmware-app-performance-data

I tried your command with exactly the same engine.conf, but I still dosn't get any data (after >60min). In vCenter I collect only level 2 statistics. But it shouldn't be of concern, because it's about the performance data, which are collected in a database (sql server).

"VMware does not recommend setting the Statistics Level higher than Level 2 unless debugging an issue. The amount of data collected is substantially greater and without adequate processing power on the SQL server could cause performance data to be not collected properly."

I found an option in vCenter advanced settings Perf.Stats.MaxCollectionThreads and have set it from default 2 to 16 in order to allow the maximum of possible synchronous connections. But it didn't help.

We plan to upgrade to esx 5.1, so I will wait, if the problem goes away after that 🙂

Thank you for your support!

[tfletcher_splun]

I just ran a test with the default configuration, trimmed it down to just the default and vc stanza.

date;perl ../bin/Engine.pm -e /home/splunkadmin/opt/splunk/etc/apps/Splunk_TA_vmware/local/test_cluster_perf.conf | grep clSvcs

Thu Jan 31 10:33:28 PST 2013
2013-01-31 18:35:00 UTC, fa=cluster_test,vc="APPS-VCENTER500", vmoid=domain-c8,meid="vc-389C22BD-5080-43F6-AB59-D71A5034107F:domain-c8", moname=Apps-SV5, perftype=clSvcs, AvgEffCpu_MHz=23817.00, AvgEffMem_MB=64407.00, perftype=clSvcs
2013-01-31 18:40:00 UTC, fa=cluster_test,vc="APPS-VCENTER500", vmoid=domain-c8,meid="vc-389C22BD-5080-43F6-AB59-D71A5034107F:domain-c8", moname=Apps-SV5, perftype=clSvcs, AvgEffCpu_MHz=23817.00, AvgEffMem_MB=64410.00, perftype=clSvcs

Despite the timestamps on the data itself it took me a long time to actually get the data, to the tune of about 20 minutes. The reason for this is that when you start up it is going to look for data from the time you started, but vCenter calculates the cluster services data based on virtual machine data that comes in so there is a significant delay in its acquisition. If you have not tried yet, do a longer running test say an hour or so either by feeding the data into splunk normally of if you prefer using the command I have provided above (i ran from the local dir hence the ../bin).

For reference my conf was:

[default]
fa = cluster_test
perfTypeWhitelist = ^cpu$|^mem$|^disk$|^net$|^clSvcs$
hostPerfCounterWhitelist =^AvgAct_KB$|^AvgConsum_KB$|^AvgGrtd_KB$|^AvgOvrhd_KB$|^AvgQueLat_ms$|^AvgRd$|^AvgRvcd_KBps$|^AvgSwpIn_KB$|^AvgSwpOut_KB$|^AvgTotLat_ms$|^AvgUsg_KBps$|^AvgUsg_pct$|^AvgVmctl_KB$|^AvgWr$|^AvgXmit_KBps$|^SumBusResets$|^SumCmdsAbort$|^SumRd$|^SumRdy_ms$|^SumWr$|^AvgSwpUsd_KB$
vmPerfCounterWhitelist =^AvgAct_KB$|^AvgConsum_KB$|^AvgGrtd_KB$|^AvgOvrhd_KB$|^AvgQueLat_ms$|^AvgRd$|^AvgRvcd_KBps$|^AvgSwpIn_KB$|^AvgSwpOut_KB$|^AvgSwpd_KB$|^AvgTotLat_ms$|^AvgUsg_KBps$|^AvgUsg_pct$|^AvgVmctl_KB$|^AvgWr$|^AvgXmit_KBps$|^SumBusResets$|^SumCmdsAbort$|^SumRd$|^SumRdy_ms$|^SumWr$|^AvgUsg_mhz$

[vcvcnum0n0]
url = https://apps-vcenter500/sdk/webService
username = svc_tfletcher
password = password
perfInstanceData = ON
interval = 5
action = PerfDiscovery
perfManagedEntityWhitelist = ClusterComputeResource|ResourcePool

Failing that you may want to look at collection level. If you'll notice I have the instance data turned on which implies a higher performance collection level. If your vCenter is configured to a low performance collection level, i.e. 1 or 2 then it may not gather requisite performance information. I am not sure this is a big risk but it may be worth looking into.

[tfletcher_splun]

For more info on perf levels and other aspects of the vcenter performance metric collection configuration check this out:

http://www.vmware.com/files/pdf/techpaper/VMware-vCenter-DBPerfBestPractices.pdf

(page 6 of the paper for info on perf levels)

[stepmmx]

Thank you for the answer. I tested it today with version 2.0 of the app, and still cannot get the data.

DRS/HA are enabled.

When I call the PerfDiscovery modul for my vCenter server, then I don't get any performance data.

perl Engine.pm -d PerfDiscovery -e /home/splunkadmin/opt/splunk/etc/apps/Splunk_TA_vmware/local/engine.conf

(the only outout is of type sourcetype=vmware:_config)

My engine.conf for the testing:

[default]

fa = Splunk-FA-ESX

[vcvcnum0n0]

url = https://vCenter-server.company.corp/sdk/webService

username = esxsplunksvc

password = xxxxxxxxxx

perfInstanceData = OFF

interval = 5

action = PerfDiscovery

perfManagedEntityWhitelist = ClusterComputeResource (the same result when I comment it out)

I get all the other data from vCenter Server: TaskDiscovery, EventDiscovery, HierarchyDiscovery etc. And I have no problem in getting performance data for the esx servers (Version esxi 4.1).

Without the data, the dashboard: "Capacity Planning for Clusters - CPU Headroom" doesn't work.

Thx in advance

[tfletcher_splun]

All scripts are modules used by Engine.pm to gather the data. PerfDiscovery.pm contains the logic for getting cluster performance data. You will notice that for this particular lookup it is looking for perftype=clSvcs - Cluster Services.

Cluster Servies data is only available for clusters with DRS/HA enabled. If you do not have those features enabled on your clusters you will never get cluster services data so that is worth checking first.

Failing that I would target 1 engine file to your vCenter with only performance discovery enabled and a perfManagedEntityWhitelist of ClusterComputeResource. In that basic configuration if you still fail to get data then let me know what you do get.