drop events from being indexed

[sreynolds30]

I have a request from some users of mine to do the following.

I need to drop events from a source and user ..

source: /var/log/uds/uds.log
user: dsapi_perftest

[niketn · Accepted solution]

@sreynolds30, one of the options you have is to search the data to be made unsearchable and run the delete command. You have to be aware that it will only make the data unsearchable and not remove from storage. Read about the delete command and understand its usage before applying.

Also before you delete existing data, you should also make sure that source uds.log is not sending data for user dsapi_perftest. If it is you should apply Regular Expression to filter out the event. Refer to documentation to filter data and send unwanted events to nullQueue before indexing.

[sreynolds30]

Sorry I guess i should have stated this better... that i just want to drop this events from being indexed and leave everything else within that source.

I'll look at the unwanted events to nullQueue

[niketn]

@sreynolds30, nullQueue will drop future events from being indexed however, you delete command was a suggestion for clearing out existing events for the user which are already indexed. Even if you do not delete, they would age out based on your index bucket rollover policy/size.

Please try out nullQueue and confirm whether you need further assistance.

[sreynolds30]

@niketn Thanks for the input.

I'm working on the nullQueue in a different test but it's not working as i think it should. Here's a sample of the logs that i don't want to index from my client from this source but just for that user.

2018-04-11T08:49:34,140 1077.dti.net [UDS] http-nio-8080-exec-25 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bWGDWVP7FJMNhjD@awAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,353 1077.dti.net [UDS] http-nio-8080-exec-46 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bAAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,552 1077.dti.net [UDS] http-nio-8080-exec-173 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bQAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,763 1077.dti.net [UDS] http-nio-8080-exec-236 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bgAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:34,989 1077.dti.net [UDS] http-nio-8080-exec-157 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="bmGDWVP7FJMNhjD@bwAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"
2018-04-11T08:49:35,196 1077.dti.net [UDS] http-nio-8080-exec-180 (com.uds.logging.RequestLogger) INFO - env="UDS-US-Stable" requestid="b2GDWVP7FJMNhjD@cAAAAC8" user="dsapi_perftest" tenantid="625c-bd25-47a5-a12d-4db5396d4ceb" role="dsapi_perftest" appid="b6de5707-cbee-477b-a332-98edce3e38e9" appname="qdxmmldbpubk" requests="1000" records="1000" status="200" method="POST" URL="/uds/batch/match/"

[niketn]

@sreynolds30, have you tried the configurations on the following line?

props.conf

[yourSourceType]
TRANSFORMS-nullQueueUnwantedUser = nullQueueUnwantedUser

transforms.conf

[nullQueueUnwantedUser]
REGEX = user\=\"dsapi_perftest\"
DEST_KEY = queue
FORMAT = nullQueue

Test using Splunk's _internal index whether events are getting dropped or not:

index=_internal sourcetype=splunkd source=*metrics.log component=metrics group=pipeline processor=nullqueue

Also, events can be dropped on indexers or Heavy Forwarders, not on Universal Forwarder.

[bhargavi]

Hi @niketn 

Could you please help me out here.  I have a little different scenario. We are integrating the json logs via HEC into Splunk Heavy Forwarder.

I have tried the below configurations.I am applying the props for the source.

In transforms, there are different regexes and I would want to route it to different indexes based on log files and route all the other files not required to a null queue. I would not be able to use FORMAT=indexqueue in transforms.conf as I cannot mention multiple indexes in inputs.conf .This is not working and no data is getting indexed. Kindly help.

The configs are like below:

PROPS.CONF --

[source::*model-app*]
TRANSFORMS-segment=setnull,security_logs,application_logs,provisioning_logs

TRANSFORMS.CONF --

[setnull]
REGEX=class\"\:\"(.*?)\"
DEST_KEY = queue
FORMAT = nullQueue

[security_logs]
REGEX=(class\"\:\"(/var/log/cron|/var/log/audit/audit.log|/var/log/messages|/var/log/secure)\")
DEST_KEY=_MetaData:Index
FORMAT=model_sec
WRITE_META=true
LOOKAHEAD=40000

[application_logs]
REGEX=(class\"\:\"(/var/log/application.log|/var/log/local*?.log)\")
DEST_KEY=_MetaData:Index
FORMAT=model_app
WRITE_META=true
LOOKAHEAD=40000

[provisioning_logs]
REGEX=class\"\:\"(/opt/provgw-error_msg.log|/opt/provgw-bulkrequest.log|/opt/provgw/provgw-spml_command.log.*?)\"
DEST_KEY=_MetaData:Index
FORMAT=model_prov
WRITE_META=true

[sreynolds30]

I got it working. Thanks for the feedback @niketnilay

[niketn]

@sreynolds30, glad you got it to work. I have converted my comment to answer. Accept to mark this as answered and upvote the comments that helped.

[logloganathan]

Could you please try this query

source= /var/log/uds/uds.log NOT "dsapi_perftest"

it will produce the event without the user from a source you mentioned

[sreynolds30]

Sorry I guess i should have stated this better... that i just want to drop this events from being indexed and leave everything else within that source.

I'll look at the unwanted events to nullQueue