How would I exclude events that normally occur after a restart of a server?

[jmaple]

On our Linux servers, we see that audit policies are re-applied to the audit service whenever the service is restarted. I'm trying to find a way to only generate events if a restart of the server has not occurred within a given period of time (likely 10-15) minutes. How would I configure my search to search for audit change events that are not within 15 minutes of a server restart?

Search for finding a change in audit settings:

index=os* key="MAC-policy" type=CONFIG_CHANGE

Search for finding server restarts:

index=os* type=SYSTEM_SHUTDOWN

[sundareshr · Accepted solution]

Let's try a different approach

index=os ("SYSTEM_SHUTDOWN" OR "CONFIG_CHANGE") | rex "(?<action>SYSTEM_SHUTDOWN|CONFIG_CHANGE)" |  autogress action as next_action | where action=next_action

[jmaple]

That does the trick.

So I understand it, are you skipping the first CONFIG_CHANGE event that occurs after a SYSTEM_SHUTDOWN event?

[sundareshr]

That's Correct

[jmaple]

That works for me. Thanks.

[sundareshr]

See if this works. What this should do, is group all events that start with Shutdown and have a CONFIG_CHANGE within 15min timespan. All other events will not be grouped. closed_txn=f, returns ungrouped events. You may have to experiment with keepevicted vs keeporphans

index=os | transaction maxspan=16m startswith="SYSTEM_SHUTDOWN" endswith="CONFIG_CHANGE" keepevicted=t | where closed_txn=0

http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Transaction

[jmaple]

Wouldn't marking all transactions as open mean that the "CONFIG_CHANGE" wasn't taking place? 100% of the time it will take place after a reboot so I need to be able to see this event if the "SYSTEM_SHUTDOWN" doesn't occur.

[sundareshr]

This doesn't mark all transaction as open. It will mark transactions as open if it DOES NOT meet any of these criteris

1) Find SYSTEM_SHUTDOWN
2) Find CONFIG_CHANGE within 15mins of SYSTEM_SHUTDOWN

Once it groups the transactions, (SYSTEM_SHUTDOWN & CONFIG_CHANGE within 15min span) remaining events will be marked as "open" transactions. The where clause will keep only the open transactions.

To make this cleaner, you could add ("SYSTEM_SHUTDOWN" OR "CONFIG_CHANGE") to your base search to minimize the events returned.

Were you able to try the query? What results did you get?

[jmaple]

The results showed transactions but the ones I saw didn't start with "SYSTEM_SHUTDOWN" so I flipped the keepevicted value to false and the events that were generated are not showing now but I need to do an effective test by rebooting my test machine then adding an audit rule.

Update: Looks like that did not produce an event when I added an audit rule. Here is what happened:

First I kept the "keepevicted=t". This produced events but the events produced are within the span of time. If the idea was to produce events outside the span of time, this did not work.

Next I changed it to "keepevicted=f". This produced no events so I changed to span down to 1s to force a result and still no result so this did not work.

Next I changed it from "keepevicted=f" to "keeporphans=t". This produced no results so it did not work.

Next I changed it to "keeporphans=f". This produced no results so it did not work.

[sundareshr]

My understanding is you DO NOT want CONFIG_CHANGE" right after a "SYSTEM_SHUTDOWN". If that's true, you should set keepevicted=t. You want all events that are NOT grouped in a transaction. To verify the findings, do a table _time _raw closed_txn and compare the events where closed_txn=0 and closed_txn=1. The one's with 1 should have "SYSTEM_SHUTDOWN" AND/OR "CONFIG_CHANGE" where as the the other should only have "CONFIG_CHANGE"

[jmaple]

I want to see the "CONFIG_CHANGE" event ONLY if the "SYSTEM_SHUTDOWN" event hasn't occurred in the last 15m. That would be the only time the "CONFIG_CHANGE" event would be relevant to our security team.