I've been trying things to figure this out for a few months now off and on. I get close but . . . and since my log output is different that anyone else's I've seen here, I'll put mine up here. I even had a Splunk instructor try this and he couldn't get it to do what I want it to do. I'm looking for: when did it happen, and how long was it down, and at some point I'd like to extend that out to: "hey this not only went down 6 hours ago, it's still down, I'll send an alert out"
Here's a chunk of the log output. I've regex'd every piece of it six ways to Sunday trying to find something to transaction or stats on, but I can't figure it out. Each line represents a hosts path on a SAN fabric so each line is actually unique but in a good SAN environment we have two of everything and there is overlap in things like switch port names, error messages etc.
So, for every host going down on a fabric message, there should be a message logged about that host on that fabric coming back up. And the same for the other fabric and no the fabrics don't know about each other (failure domain seperation etc.)
99% of the time, I don't care about the activity at all unless it looks like the host fell away on one fabric and not the other and then never came back. That's what I want to know about.
12/6/18
4:50:04.000 AM
Dec 6 04:50:04 UNIQUE-SWITCH-NAME-1 : 2018 Dec 6 04:50:05 CST: %PORT-5-IF_UP: %$VSAN 11%$ Interface fc1/35 is up in mode F hostname-XXXXX-fabric1
12/6/18
4:50:02.000 AM
Dec 6 04:50:02 UNIQUE-SWITCH-NAME-2: 2018 Dec 6 04:50:03 CST: %PORT-5-IF_UP: %$VSAN 12%$ Interface fc1/35 is up in mode F hostname-XXXXX-fabric2
12/6/18
4:50:04.000 AM
Dec 6 04:50:04 UNIQUE-SWITCH-NAME-3 : 2018 Dec 6 04:50:05 CST: %PORT-5-IF_UP: %$VSAN 11%$ Interface fc1/35 is up in mode F hostname-ZZZZZ-fabric1
12/6/18
4:50:02.000 AM
Dec 6 04:50:02 UNIQUE-SWITCH-NAME-4: 2018 Dec 6 04:50:03 CST: %PORT-5-IF_UP: %$VSAN 12%$ Interface fc1/35 is up in mode F hostname-ZZZZZ-fabric2
12/6/18
4:47:14.000 AM
Dec 6 04:47:14 UNIQUE-SWITCH-NAME-1 : 2018 Dec 6 04:47:15 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 12%$ Interface fc1/35 is down (Link failure loss of signal) hostname-XXXXX-fabric1
12/6/18
4:47:13.000 AM
Dec 6 04:47:13 UNIQUE-SWITCH-NAME-2 : 2018 Dec 6 04:47:14 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 11%$ Interface fc1/35 is down (Link failure loss of signal) hostname-XXXXX-fabric2
12/6/18
4:47:13.000 AM
Dec 6 04:47:14 UNIQUE-SWITCH-NAME-3 : 2018 Dec 6 04:47:13 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 12%$ Interface fc1/36 is down (Link failure loss of signal) hostname-ZZZZZ-fabric1
12/6/18
4:47:13.000 AM
Dec 6 04:47:13 UNIQUE-SWITCH-NAME-4 : 2018 Dec 6 04:47:14 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 11%$ Interface fc1/36 is down (Link failure loss of signal) hostname-ZZZZZZ-fabric2
... View more