Alerting

Monitoring Splunk logs for alert delete

snehal8
Path Finder

Hello All,

We have two search head, three indexers clustered, a cluster master, and a deployment server. All running Windows 2k8 R2.

We are finding some alerts are deleted form search head, and need to investigate log files to monitor which user deleted, when and so on.

Cloud you please guide me how to figure out this?

Thanks.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi snehal8,

run this search:

index=_audit host=YourHostName action=alert_deleted

the result will look like this:

Audit:[timestamp=02-12-2015 10:39:21.783, user=TheBadGuyHowDeletedTheAlert, action=alert_deleted, sid="scheduler__AnyUserName_REFfUkNQX0xEQVA__RMD5d4292166408c9a03_at_1423733700_18910", trigger_time=1423733705, deleted=1][n/a]

Hope this helps ...

cheers, MuS

lmyrefelt
Builder

Remember to check so you are not over-writing any files under local or similar ... if you are seemingly "loosing" data , searches, alerts, views, whatever ... this can happen thru the forwarder-managment / deployment server / cluster-deploy tools that are in use.

0 Karma

aakwah
Builder

Hello,

All audit information are saved on audit.log /opt/splunk/var/log/splunk/audit.log audit logs are indexed in _audit index, you can search them index=_audit and create alerts on search queries you want.

Regards

snehal8
Path Finder

Thanks for reply @aakwah, but when i searched for index=_audit "mysearch name" "*delete*" its displaying log for my this search string, please guide me if am going wrong ?

0 Karma

karlbosanquet
Path Finder

https://answers.splunk.com/answers/43339/alert-on-deleted-data.html

Specifically;

Put a crazy string in your search, like so:

index=_audit "action=search" search=*delete* NOT dfsdkljwkwtw
This will prevent your search from showing up in the results.

You might want to refine it a big using a regex to look for | delete, |delete, | delete , etc.

0 Karma

aakwah
Builder

Welcome, you can make use of action field to specify a certain action you want to look for,

index=_audit action=delete

Some of avialble actions:

index=_audit | stats count by action | table action

CREATE_PASSWORD
EDIT_PASSWORD
GET_PASSWORD
REMOVE_PASSWORD
accelerate_datamodel
accelerate_search
add
delete
edit_roles
edit_server
edit_user
embed_report
license_edit
list_inputs
login attempt
quota
read_session_token
rest_properties_get
rest_properties_set
restart_splunkd

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...