The input is working and the events are getting to Splunk. I am trying to get a filter going to drop noisy events. I have created an app that is deployed via a configuration server. I have tried this many different ways, but it doesn't seem to drop the events. The app is being pushed to the Universal Forwarder on Windows. The regex matches on online tools.
props.conf
[SAP_portal_security_audit]
TRANSFORMS-set = discard_events
transforms.conf
[discard_events]
REGEX = (ACCESS\.ERROR|USERMAPPING\.USE)
DEST_KEY = queue
FORMAT = nullQueue
Log data
.0#2016 06 29 08:54:55:906#0-500#Info#/System/Security/Audit/UserMapping#
#BC-JAS-SEC-UME#com.sap.security.core.sda#C0000A980ACE0ACF0000000600003AE4#28973850000000004#sap.com/irj#com.sap.security.core.util.SecurityAudit#Guest#0##90B5B81E3DA611E6B986000001BA1B1A#90b5b81e3da611e6b986000001ba1b1a#90b5b81e3da611e6b986000001ba1b1a#0#Thread[pool-2141-thread-1,5,Dedicated_Application_Thread]#Plain##
User mapping used | USERMAPPING.USE | USER.PRIVATE_DATASOURCE.un:eServices | | systemtype=[SAP_CRM], system=["SAP_CRM" (system landscape: "EnterprisePortal")], remote user ID=[ZESRVUSER], uses strong encryption=[true]#
#2.0#2016 06 29 10:41:50:911#0-500#Warning#/System/Security/Audit/Access#
#EP-KM-FWK-RF#sap.com/com.sap.netweaver.bc.rf#C0000A989048C7010000000400000EA0#2778350000000004#sap.com/eServicesMasthead#com.sapportals.wcm.repository.security.SecurityAudit$AccessLog#UKSHEPA#16689##FCBF8AB33E0F11E6CA770000002A64EE#fb491e783e0f11e6c3fb0000002a64ee#fb491e783e0f11e6c3fb0000002a64ee#0#Thread[1047252450|pcd:portal_content/bungeContent/protectedContent/authenticatedDesktop/frameworkPages/authenticatedFrameworkPage/eServicesLightMasthead\#com%2esap%2eportal%2enavigation%2eportallauncher%2edefault.pcd%3aportal_content%2fbungeContent%2fprotectedContent%2fauthenticatedDesktop%2fframeworkPages%2fauthenticatedFrameworkPage.eServicesLightMasthead,5,Managed_Application_Thread]#Plain##
UKSHEPA | ACCESS.ERROR | /documents/Public Documents/AgSite/SiteImages/b4b99ce4da58004ef7e8614edb99e3d2.xml | leaf_write_content,leaf_write_properties#
6/29/16
11:03:07.943 AM
#2.0#2016 06 29 11:03:07:943#0-500#Warning#/System/Security/Audit/Access#
#EP-KM-FWK-RF#sap.com/com.sap.netweaver.bc.rf#C0000A9890453995000000CE00003A30#6820051000000004#sap.com/eServicesPublic#com.sapportals.wcm.repository.security.SecurityAudit$AccessLog#Guest#0##F59B7BD13E1211E6AE430000006810D3#f59b7bd13e1211e6ae430000006810d3#f59b7bd13e1211e6ae430000006810d3#0#Thread[1784794969|pcd:portal_content/bungeContent/publicContent/roles/eServices_Home/publicHome/locationInfoTabbedContainer/PersonnelSummary\#com%2esap%2eportal%2enavigation%2eportallauncher%2eanonymous.pcd%3aportal_content%2fbungeContent%2fpublicContent%2fanonymousLightDesktop%2fframeworkPages%2fanonymousLightFramework.com%2esap%2eportal%2elightinnerpage.com%2esap%2eportal%2elightcontentarea.content.locationInfoTabbedContainer.PersonnelSummary,5,Managed_Application_Thread]#Plain##
Guest | ACCESS.ERROR | /documents/Public Documents/AgSite/Personnel/9f40e230b555f773b47ffb300514e66e.xml | leaf_write_content,leaf_write_properties#
Hi tomcochran,
i guess you have deployed the configuration above on your forwarder? If so, thats the problem (i assume your configuration is correct) !
The UniversalForwarder does not parse any of the events he is forwarding. Just bring the configuration to your indexer and it should work.
I hope it helps.
Greetings
Thank you, this worked. So here is a question regarding License, Do the events getting dropped at the indexer count towards the capacity? Is it capacity getting to the indexer, or capacity of data indexed?
Hi tomcochran,
i'm glad i could help. I made my comment an answer. Can you do me a favor and accept it?
To answer your license questions, it's capacity of data indexed. The events send to nullqueue wont stress your licence.
Greetings
hgrow
Awesome, thank you so much.
I solved a similar issue configuring two stanzas in transforms.conf and inserting two commands in props.conf.
note that it's important the order between the two commands in props.conf (the order of stanzas in transforms.conf isn't important):
in other words: before the command with all the logs (REGEX=.) and after the command with the set of logs you want (REGEX=xxxx).
see the following example where set_AS are the logs I want to index, nullqueue are the logs I want to discard.
in props.conf
TRANSFORMS-set-AS=set_nullqueue,set_AS
in transforms.conf
[set_nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue
[set_AS]
REGEX=SRVE
DEST_KEY = queue
FORMAT = indexQueue
Bye.
Giuseppe
I have tried
props.conf
[hfss_source]
TRANSFORMS-set= set_parsing,set_null
transforms.conf
[set_null]
REGEX = standby
DEST_KEY = queue
FORMAT = nullQueue
[set_parsing]
REGEX =.
DEST_KEY=queue
FORMAT=indexQueue
Trying to get rid of all entries with the word "standby" in it, but everything gets indexed.
splunk cmd btool props list hfss_source
shows correctly, but
splunk cmd btool transforms list hfss_source
shows absolutely nothing
Any ideas ? I am completely lost, I feel I have done what everybody is writing will work, but I just can't get it to work?
Any help would be greatly appriciated
Hi tomcochran,
i guess you have deployed the configuration above on your forwarder? If so, thats the problem (i assume your configuration is correct) !
The UniversalForwarder does not parse any of the events he is forwarding. Just bring the configuration to your indexer and it should work.
I hope it helps.
Greetings