how to blacklist particular REST API events from being indexed into Splunk's main index?

[Hemnaath]

Hi All, We have a request from a user to disable the events that are coming from the source="rest://Solarwinds Nodes". These events are extremely large and consume unnecessary disk space (every 5 minutes) and licensing. it appears to be a REST call originating on host1. We are getting the events when we execute the below query in search head.

Query details :

host=host1* source = rest://Solarwinds Nodes sourcetype = rest:solarwinds:nodes 

Events details :

{[-]

{"results":[{"solarwinds_node_id":2,"polling_engine_id":12,"polling_engine":"VMTP01","solarwinds_prefix":"N:","src_ip":"10.X.X.X","host":"","percent_memory_used":66,"cpu_load":12,"up_since":"2016-03-02T16:52:00","host_tier":null},{"solarwinds_node_id":3,"p.....


]
}

Show as raw text

Inputs.conf details : path :/opt/splunk/etc/apps/search/local

[rest://Solarwinds Nodes]
auth_type = none
endpoint = https://ws.xxxx.com/sw/getnodes
http_method = GET
index = main
index_error_response_codes = 0
polling_interval = 300
response_type = json
sourcetype = rest:solarwinds:nodes
streaming_request = 0

Question :

1) How to blacklist these events without getting indexed into main? I mean what values should be set as blacklist = ?
example : Should I blacklist based on hostname or source or source type?
2) This particular inputs.conf is present in the deployment/license manager instance, on changing the configuration, should I need to restart the Splunk service or we need to execute ./splunk reload deploy-server?

Any assistance would be greatly appreciated.

[somesoni2]

Instead of blacklisting the event from ingestion, it would be better to just turn off that input. Either comment OR add disabled = 1 in the inputs.conf entry. If it was deployed from Deployment server, make the necessary change and do reload (generally it sufficient). Wait for some time and check if the Clients have received the updated inputs.conf. If not restart Splunk service on Deployment Server.

[Hemnaath]

somesoni, is it possible to tell me how to blacklist the particular events based on multiple hostname or source type or source.

Any assistance would be greatly appreciated.

[somesoni2]

I believe what you want to do is event filtering. It is one of the common things Splunkers do to keep Splunk clean of junk data. See these

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Routeandfilterdatad#Discard_specific_ev...
https://answers.splunk.com/answers/107605/filtering-events-out-via-props-conf-and-transforms-conf.ht...

[Hemnaath]

thanks somesoni2, above events are not getting into splunk after disabling and restarting the splunk service. In Portal when searched for this events we are not getting any result, after disabling and restarting the splunk services.

solution : adding disable = 1 stanza into the inputs.conf and restarting the splunk service resolved the issue

[rest://Solarwinds Nodes]
auth_type = none
disable = 1
endpoint = https://ws.xxxx.com/sw/getnodes
http_method = GET
index = main
index_error_response_codes = 0
polling_interval = 300
response_type = json
sourcetype = rest:solarwinds:nodes
streaming_request = 0

[Hemnaath]

thanks Somesoni, for providing some inputs on this issue but not sure about the command which you had mentioned.

cmd
path /opt/splunk/bin
./splunk _internal call /configs/conf-inputs/_reload

thanks in advance.

[somesoni2]

The command is for reloading inputs configuration without a restart. For it doesn't work all the time and hence I asked you to run it and test. Last resort will be to reload deployment server (or restart).

[Hemnaath]

thanks Somesoni, this particular input configuration is present under this path.
Inputs.conf details : path :/opt/splunk/etc/apps/search/local

[rest://Solarwinds Nodes]
auth_type = none
endpoint = https://ws.xxxx.com/sw/getnodes
http_method = GET
index = main
index_error_response_codes = 0
polling_interval = 300
response_type = json
sourcetype = rest:solarwinds:nodes
streaming_request = 0

But we do not have the GUI access to disable this setting , so we had directly edited the inputs.conf file but did not execute splunk restart or reload command after making the changes in configuration file. So kindly let me know whether we need to restart the service/reload.

thanks in advance.

[somesoni2]

If you're updating the inputs.conf file directly, you need to restart the splunk instance for it to take effect. Before that, could you try running this reload command and check if the data input is disabled (no data in being ingested by it)

$Splunk_Home/bin/splunk _internal call /configs/conf-inputs/_reload 

[Hemnaath]

thanks Somesoni2 for throwing some lights on this but initially I had tried to do disabled = 1 but did not execute ./splunk reload deploy-server as this configuration itself present in the deployment manager instance. So kindly let me know whether should I need to execute the reload deploy-server command and also let us know how to blacklist the particular events based on host/source type/source.

thanks in advance.

[somesoni2]

Question, where on deployment server (DS) you've this inputs.conf entry, under $Splunk_home/etc/apps OR $Splunk_home/etc/deployment-apps? If it's former (data input is actually created on DS), you can log into Splunk web UI and disable it from Settings->Data Inputs page. If it's in later (deployment-apps), you need to update the inputs.conf to add disabled, and run the reload command, for it to go to the deployment-client where it was running.