Getting Data In

What am I missing in this props/transforms pair to filter "noise" from my log?

rgcurry
Contributor

I am trying to filter log "noise" before the data gets indexed but the filtering is not working. I have tested the REGEX in the transforms.conf in a search, it works perfectly. But the noise is coming thru and I am not seeing what I have missed. I've dug thru probably 2 dozen other questions/inquiries here with similar situations but none of those answers has solved my situation. Perhaps your review will pick up what I am not.

My props.conf file looks like this:

[log_data]
TRANSFORMS-set = dumpNoise,keepInfo 
EXTRACT-filename = loaded\"\s(?<filename>\S+) 
EXTRACT-course_filedate = coursefiledate\w(?<course_filedate>\w+) 
EXTRACT-transcript_filedate = transcriptfiledate\w(?<transcript_filedate>\w+)

"log_date" is the correct sourcetype for this data coming into the Indexer.

The transforms.conf file looks like this:

[dumpKeepAlives]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keepFileXferInfo]
REGEX = action executed|action failed|file upload|file download|trigger error
DEST_KEY = queue
FORMAT = indexQueue

Here is a sample of the data in the log that I am tring to filter:

2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:42 192.168.71.22 24504 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:43 - - sftp-stg.bazaarvoice.com 22 - - - "external file uploaded" /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catal
og.xml 4857550 -^M
2012-07-16 17:08:43 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.SftpRegexFileUploadAction; message=files which matches with .* has been uploaded" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.MoveRegexFileAction; message=the file(s) /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/outbound/bv_catalog.xml
 has been moved" - -^M
2012-07-16 17:08:44 - - - - - - - "action executed" "trigger=BazaarVoiceOutbound_LENA288_sftp-stg.bazaarvoice.com_ENTMFILE_3_4_DigitalPlatform; class=class c
om.jscape.inet.mft.workflow.actions.DeleteFileAction; message=file /jscape/JSCAPE_MFT_Server/users/ExternalUsers/BazaarSFTP/trigger/Done deleted" - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:47 192.168.71.22 36487 192.168.64.188 22 - - - "session closed; " - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session started" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - SSH-2.0-JSCAPE - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "logged out" - - -^M
2012-07-16 17:08:52 192.168.71.22 27319 192.168.64.188 22 - - - "session closed; " - - -^M

Everything is coming thru, the noise and the wanted data. What do you see that I am missing?

I appreciate any suggestions on this.

0 Karma

Ayn
Legend

In props.conf, you refer to the transforms dumpNoise and keepInfo, but in transforms.conf the names are dumpKeepAlives and keepFileXferInfo, so obviously that won't match. Typo in your post here or typo in your conf files as well?

rgcurry
Contributor

OOPS! Typo in my post. The values in the transforms.conf is what I am really using in both on the system. I plan to change the names as they reflect more what I am trying to do now. I was filtering a smaller set of data before, which was working (and still working) fine.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...