Getting Data In

Filter portions of multi-line logs using a Heavy Forwarder

Raghav2384
Motivator

Hello Experts, I had posted the same question couple of days ago and had to re-post because of the formatting issues. We are consuming certain logs with DEBUG level but realized only a portion of those we need to index. We have decided to extract those required portions and discard rest of the stuff. These are logs averaging 50 lines per event. I just want to make sure we do it right the first time as the usage is pretty deep and do not want to interrupt. Here's a sample log

172.18.232.85 [2015-03-04 15:20:25,083] ===============================================
POLICY RESULT ERROR: Unable to authorize session - no authorization result found
Key1 = value1
Key2 = Value2
Key3 = Value3
Key4 = Value4
Key5 = Value5
Key6 = Value6
Key7 = Value7
Key8 = Value8
Key9 = Value9
DEBUG MSGS:

INFO : (core) Tagging message with ID: Important Stuff here
INFO : (radius) RADIUS device group assigned to: Phoenix
INFO : (radius) Loading session by the audit session id: XYZ
INFO : (core) Lock obtained on key: auditSessionId:ab1234bcngd
INFO : (core) Start session triggered
INFO : (radius) Radius usage reported 123456
INFO : (radius) Found generic location parameter ssid/DomainID
INFO : (radius) Found generic location parameter ap_mac/12-23-45-67-89
INFO : (location) Using generic ssid\XYZABC for location lookup.
INFO : (location) Using generic ap_mac\12-23-45-67-89 for location lookup.
INFO : (location) Location found for generic matching: ssid\DomainID
WARN : (auth) Failed USUM_AUTHORIZATION no password found for user
WARN : (core) Removing session since no authorization result found
WARN : (service) Stopping creation since the session has no services
INFO : (balance) Error found, rolling back transaction

ERROR : (core) Error processing policy request: Unable to authorize session - no authorization result found

we do not want to index entire Debug...I just need to grab the fields/Text in Bold and send rest to the null Queue. All of these are multiline logs and i am not able to get close in achieving this. Appreciate any pointers

Thanks,
Raghav

0 Karma

woodcock
Esteemed Legend

You can do it like this in your props.conf:

[MyBigFatSourcetype]
SEDCMD-slimfast = s/^INFO : (core) Start.*$// s/^INFO : (radius) Radius.*$// s/^INFO : (location).*$// s/^WARN :.*$// s/^INFO : (balance).*$//

This will definitely work but the downside is that, although the text will definitely be gone, I think it will leave blank line gaps in the raw events which may be distracting/confusing.

Raghav2384
Motivator

Any ideas?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Is the format of the logs always same?? We need to identify patterns for which lines to retain and which lines to discard.

0 Karma

Raghav2384
Motivator

Yes, standard is, the pieces i want from debug portion remains same

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...