How do I monitor and alert on any success or failure of a su to root on my *nix systems?
You can (if not already) monitoring for sudo logs in your nix systems. I believe it should be available at /var/adm/sudo.log. Once the data is available in Splunk, you can use following to learn how to create alerts.
http://docs.splunk.com/Documentation/Splunk/6.2.0/Alert/Aboutalerts www.splunk.com/view/SP-CAAAGYG
If you can query those events in search you can then create an alert on those.
In top right of window click "Save As" and select "Alert" from the drop down. It will walk you through it.
This could help if you haven't seen it yet. http://docs.splunk.com/Documentation/Splunk/6.3.2/Alert/Aboutalerts
What would this search query look like?
On CentOS the su failure is logged in /var/log/messages like this:
/var/log/messages
Jul 19 12:38:17 localhost su: FAILED SU (to root) non_root on pts/0
So something like this query:
index=whatever_your_index_is "FAILED SU (to root) "
