Several thousand remote hosts connecting to a collector at 2 geographically distanced datacenters.
One of the collectors should be getting bytes_seen > zero
If neither collector sees bytes_seen > zero, then the controller is assumed to be down and an alert needs to be raised.
Trying to table
RemoteDevice DatacenterCollector BytesSeenWestCoast DatacenterCollector BytesSeenEastCoast RemoteDeviceStatus
Device-1 Collector-WestCoast 50 Collector-EastCoast 0 Up
Device-2 Collector-WestCoast 104 Collector-EastCoast 0 Up
Device-3 Collector-WestCoast 0 Collector-EastCoast 0 Down
In this scenario, Device-3 should be marked down, and an alert generated.
Expecting to use a lookup table file for a list of all the remote devices Splunk needs to receive the events for from the datacenter collectors.
try this
-------- | table RemoteDevice DatacenterCollector BytesSeenWestCoast DatacenterCollector BytesSeenEastCoast | eval RemoteDeviceStatus=if(BytesSeenWestCoast=0 AND BytesSeenEastCoast=0,"Down","Up") | table RemoteDevice DatacenterCollector BytesSeenWestCoast DatacenterCollector BytesSeenEastCoast RemoteDeviceStatus | where RemoteDeviceStatus like "Down"
Hi. I don't know what your data looks like but it might be something like
<search to get your logs...>
| stats latest(CollectorWestCoastCount) AS WestCoastCount, latest(CollectorEastCoastCount) AS EastCoastCount by RemoteDevice
| search WestCoastCount=0 AND EastCoastCount=0
And then alert when you have count > 0
If you show your log samples we can refine from here
To be less cryptic. . .
d1xhop01(CollectorWestCoast) and d2xhop01 (CollectorEastCoast)
RemoteDevice is the store controller. In this case: "CC4001-(Store 4001 CC Controller Div 35)"
Here is a sample message seen from d2xhop01 (CollectorEastCoast) that was forwarded to Splunk.
The challenge is to compare bytes_seen from both collectors at the same time for each RemoteDevice and alert if bytes_seen is zero from both collectors for the period defined.