Deployment Architecture

How can Splunk ignore data after it matches with some characters?

subaldhawan
New Member

I have a log file in below format:

Section 1
a1 a
a2 b

Section 2
axx abc
aff def

I want to ignore all raw data coming after "Section 2" from log file before indexing it into Splunk. Is there any way we can do this in Splunk?

Kindly note only universal forwarder is configured.

Tags (1)
0 Karma

micahkemp
Champion

There is no way to do this if you want

a1 a
a2 b

to be separate events.

The only way I can think of something that even comes close to what you want is to define your LINE_BREAKER and linemerging options such that the entire sections are single events, then using a TRANSFORM at index time to nullQueue the section2 event all at once. But again, this means that all of section1 would be one large event, and I can't imagine that's what you want.

This seems like a prime case for pre-data massaging before sending to splunk.

0 Karma

Elsurion
Communicator

Is this a normal logfile which will contain Section 1 and Section 2 more then once?
e.g.

Section 1 --> Section 2 --> Section 1 --> Section 2 --> Section 1?

If yes you can forward your Data to a so called Heavy Forwarder. Which is a full installation of Splunk>. On this instance you can filter your data before you index it.
https://docs.splunk.com/Documentation/Splunk/6.6.3/Forwarding/Deployaheavyforwarder

As example a stip of lines containing a special keyword.
Setup for transforms.conf

[setnull]
REGEX = \s;\sDefault\s;\s
DEST_KEY = queue
FORMAT = nullQueue

Setup for props.conf

[mysourcetype]
TRANSFORMS-null= setnull

To get this thing working, we should know now how the ignored part can be selected for deletion.

0 Karma

subaldhawan
New Member

No, Section 1 and 2 will come only once.

And the no of rows within Section 1 and 2 are dynamic.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's something a universal forwarder cannot do.

---
If this reply helps you, Karma would be appreciated.
0 Karma

subaldhawan
New Member

Thank you. Can you tell if there is an option at indexer side.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...