Monitoring Splunk logs for alert delete

[snehal8]

Hello All,

We have two search head, three indexers clustered, a cluster master, and a deployment server. All running Windows 2k8 R2.

We are finding some alerts are deleted form search head, and need to investigate log files to monitor which user deleted, when and so on.

Cloud you please guide me how to figure out this?

Thanks.

[MuS]

Hi snehal8,

run this search:

index=_audit host=YourHostName action=alert_deleted

the result will look like this:

Audit:[timestamp=02-12-2015 10:39:21.783, user=TheBadGuyHowDeletedTheAlert, action=alert_deleted, sid="scheduler__AnyUserName_REFfUkNQX0xEQVA__RMD5d4292166408c9a03_at_1423733700_18910", trigger_time=1423733705, deleted=1][n/a]

Hope this helps ...

cheers, MuS

[lmyrefelt]

Remember to check so you are not over-writing any files under local or similar ... if you are seemingly "loosing" data , searches, alerts, views, whatever ... this can happen thru the forwarder-managment / deployment server / cluster-deploy tools that are in use.

[aakwah]

Hello,

All audit information are saved on audit.log /opt/splunk/var/log/splunk/audit.log audit logs are indexed in _audit index, you can search them index=_audit and create alerts on search queries you want.

Regards

[snehal8]

Thanks for reply @aakwah, but when i searched for index=_audit "mysearch name" "*delete*" its displaying log for my this search string, please guide me if am going wrong ?

[karlbosanquet]

https://answers.splunk.com/answers/43339/alert-on-deleted-data.html

Specifically;

Put a crazy string in your search, like so:

index=_audit "action=search" search=*delete* NOT dfsdkljwkwtw
This will prevent your search from showing up in the results.

You might want to refine it a big using a regex to look for | delete, |delete, | delete , etc.

[aakwah]

Welcome, you can make use of action field to specify a certain action you want to look for,

index=_audit action=delete

Some of avialble actions:

index=_audit | stats count by action | table action

CREATE_PASSWORD
EDIT_PASSWORD
GET_PASSWORD
REMOVE_PASSWORD
accelerate_datamodel
accelerate_search
add
delete
edit_roles
edit_server
edit_user
embed_report
license_edit
list_inputs
login attempt
quota
read_session_token
rest_properties_get
rest_properties_set
restart_splunkd