WMI intermittent issues / missing data

[MrNetworker]

Hi Everyone,

I have setup my Splunk indexer and a few Universal forwarders to poll the performance stats of the individual machines via WMI and send the data to the index machine. The event logs are also sent however I have not seen any issues with function so far. Everything is working well however the WMI stats for CPU etc will fail every so often and data will be lost for an hour.
The errors in splunkd.log seem to match up to the missing data.

The box's which are having the issues are running Server 2003 Standard R2.
I also have a box on XP Pro with the same config files and it does not seem to have any issues. All servers are generating the same "800706BF" error.

Please see the below dumps of the files:
Any help would be appreciated.

Splunkd.log:

07-22-2012 22:25:28.037 +1200 INFO TcpOutputProc - Connected to idx=10.1.1.0:9997 using ACK.

07-22-2012 22:25:28.255 +1200 INFO TcpOutputProc - Connected to idx=10.1.1.0:9997 using ACK.

07-22-2012 23:33:58.720 +1200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe" -index windows" WMI - Error occurred while trying to retrieve results from a WMI query (error="The remote procedure call failed and did not execute." HRESULT=800706BF) (\localhost\root\cimv2: Select PercentProcessorTime,PercentUserTime from Win32_PerfFormattedData_PerfOS_Processor where Name = "_Total")

07-23-2012 02:37:07.759 +1200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe" -index windows" WMI - Error occurred while trying to retrieve results from a WMI query (error="The remote procedure call failed and did not execute." HRESULT=800706BF) (\localhost\root\cimv2: Select PercentProcessorTime,PercentUserTime from Win32_PerfFormattedData_PerfOS_Processor where Name = "_Total")

07-23-2012 05:36:56.246 +1200 INFO WatchedFile - Will begin reading at offset=24992167 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1'.

07-23-2012 05:36:56.558 +1200 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.

07-23-2012 05:49:08.737 +1200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe" -index windows" WMI - Error occurred while trying to retrieve results from a WMI query (error="The remote procedure call failed and did not execute." HRESULT=800706BF) (\localhost\root\cimv2: Select FreeMegabytes,Name,PercentFreeSpace from Win32_PerfFormattedData_PerfDisk_LogicalDisk)

07-23-2012 08:15:09.640 +1200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe" -index windows" WMI - Error occurred while trying to retrieve results from a WMI query (error="The remote procedure call failed and did not execute." HRESULT=800706BF) (\localhost\root\cimv2: Select CurrentDiskQueueLength,Name,PercentDiskReadTime,PercentDiskTime,PercentDiskWriteTime,DiskBytesPerSec from Win32_PerfFormattedData_PerfDisk_PhysicalDisk)

WMI.conf:
[WMI:CPUTime]
index = perfmon
server = localhost
wql = Select PercentProcessorTime,PercentUserTime from Win32_PerfFormattedData_PerfOS_Processor where Name = "_Total"
interval = 3
disabled = 0

[WMI:FreeDiskSpace]
index = perfmon
server = localhost
wql = Select FreeMegabytes,Name,PercentFreeSpace from Win32_PerfFormattedData_PerfDisk_LogicalDisk
interval = 120
disabled = 0

[WMI:LocalNetwork]
index = perfmon
server = localhost
wql = Select CurrentBandwidth,Name,BytesReceivedPerSec,BytesSentPerSec,BytesTotalPerSec from Win32_PerfFormattedData_Tcpip_NetworkInterface
interval = 10
disabled = 0

[WMI:LocalPhysicalDisk]
index = perfmon
server = localhost
wql = Select CurrentDiskQueueLength,Name,PercentDiskReadTime,PercentDiskTime,PercentDiskWriteTime,DiskBytesPerSec from Win32_PerfFormattedData_PerfDisk_PhysicalDisk
interval = 10
disabled = 0

[WMI:LocalProcesses]
index = perfmon
server = localhost
wql = Select IDProcess,Name,PercentProcessorTime,PrivateBytes from Win32_PerfFormattedData_PerfProc_Process
interval = 30
disabled = 0

[WMI:Memory]
index = perfmon
server = localhost
wql = Select AvailableMBytes,CommittedBytes,PercentCommittedBytesInUse,PagesPerSec from Win32_PerfFormattedData_PerfOS_Memory
interval = 5
disabled = 0

Regards.

[Drainy]

At a quick first glance it looks like it is more likely to be a Windows WMI issue instead of a Splunk one. Have a look at; http://www.microsoft.com/en-us/download/details.aspx?id=7684

It is a diag tool released by Microsoft to help identify problems with WMI on host systems.