tracking hosts through a state diagram

[jrstear]

I need to track hosts as they move through a state diagram. Specifically, show only events that indicate state changes. Please help on the below approach, or suggest an alternate.

Say there are only two states, X and Y. Eventtype=X-Y matches events which trigger a state transition from X to Y, and similarly for eventtype=Y-X. These eventtypes are set as priority 1, so they appear first if the event matches multiple eventtypes. They also all get tag=state, such that a search for tag=state yields all events which might indicate a state change. The below finds only the state changes:

tag=state | sort +_time | eval from=substr(eventtype,1,3) | eval to=substr(eventtype,-3) |
  streamstats last(to) AS lastto BY host current=false window=1 global=false  |
  where from=lastto OR isnull(lastto)

The isnull results in the first seen event triggering a transition (no previous knowledge of state).

However, what if there are more than two states, say X, Y, and Z? Now instead of simply omitting sequential events with matching eventtypes (the above search could be simplified but I was setting the stage), the actual state of the hosts has to be tracked, such that an X-Y event only triggers a transition to Y if the host is in X (omit the event as before if the host is already in Y, but also if it is in Z). Can this be done with the splunk search language?

We wrote a custom command to do this, but it is unacceptably slow. So I am pursuing a search language approach per the race results at http://splunk-base.splunk.com/answers/5653/slow-running-custom-search-command (and this command seems inherently non-streaming). http://www.cs.sandia.gov/~jrstear/ras/ gives more info on the actual application.

[jrstear]

We rewrote the python script, which now runs acceptably fast.

[mw]

I'm not completely sure I'm following, but you could collapse fields and use regular expressions in order to detect state change. As an example, I can detect systems which have recently come online via something like this:

sourcetype=ping_scan | stats list(status) as statuses first(_time) as _time by dest_ip, dest_host | eval statuses = mvjoin(statuses, " ") | eval alert = if(match(statuses, "^Down .* Up"), "True", "False") | where alert == "True"

Basically, take multiple "status" values, collapse them to a single string which looks like "Up Up Down Down Up", and then just match against that. Obviously, the regular expression can be as complex as needed (a regular expression literally being a state machine).

Hope that helps. Credit to gkanapathy for making this possibility known to me in the first place.

[jrstear]

Thanks, but I don't think that is sufficient either. Each incoming event may justify a state change for the host, depending on the state of the host when the event is received. So the state of the host after the previous event) must be known in order to decide if the current event causes a change. A hash with host as key and state as value would be great, which persists across all events.

[jrstear]

Using streamstats seems insufficient, because the state decision needs to be made prior to streamstats (so the state can be passed downstream).

decision(needs info about previous event) -> streamstats(provides info about previous event)

Please correct if wrong, or provide other ideas. Thanks.