Splunk Search

Each File as One Single Splunk Event

jefferson_santa
Engager

Hi everyone,

I need solve a issue as simple as that: my system generate many files and each file is a isolated event.

Each file has many lines (more than 700 lines) but to my business each file is just one single event.

How configure Splunk to treat each file as a single event?

Thanks,

Jefferson Santana

Tags (2)

ss026381
Communicator

I am using splunk plugin in Jenkins. Where would I make change so that Splunk consider Jenkins log file as one event? I do not have access to .conf files.

If I have to change in .conf file, I may ask admin to make this change but I don't know what change I have to make. Help is appreciated. ,I am using splunk plugin in Jenkins to send Jenkins logs to the Splunk. I want Splunk to treat one log file as a single event. Where would I use ((?!)) or ((*FAIL)) to achieve this? Do I have to make changes to prop.conf and input.conf? What if I do not have access to those files on Splunk server?

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@ss026381 - This question you left a comment on is quite old and may not generate much activity. I would recommend asking a new question. Thank you!

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Any way to just move that ss026381 comment to a new question? It seems pretty complete, but it's seemingly unrelated to where ss026381 posted it.

0 Karma

ss026381
Communicator

Yea I created new question. Thanks guys

0 Karma

dwaddle
SplunkTrust
SplunkTrust

One way is to set up a dummy/impossible LINE_BREAKER.

In inputs.conf

[monitor:///path/to/files]
sourcetype=my_system

In props.conf (on indexer if using universal forwarder):

[my_system]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]*)-=-=-=-=ThIs-iS-An-ImPoSsiBle-StRiNg=-=-=-=-

If these files change, you may want to also set the CHECK_METHOD on the forwarder itself.
In props.conf (on the forwarder):

[source::/path/to/files/...]
CHECK_METHOD=entire_md5

gkanapathy
Splunk Employee
Splunk Employee

The LINE_BREAKER you want is actually either ((?!)) or ((*FAIL)), both of which are guaranteed to fail regardless of the content of your input file.

gkanapathy
Splunk Employee
Splunk Employee

The easiest and most efficient way is to set a single sourcetype for your file, and define the rules for this sourcetype:

[mysinglefilesourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((*FAIL))
TRUNCATE = 99999999

This disables line-merging, which sounds wrong, but in fact, you don't want or need it since you won't be breaking the file into separate lines in the first place. The specified LINE_BREAKER is a special PCRE regex that will never break on any line the file, guaranteed. The TRUNCATE setting is there to make sure the entire file is counted as the event, because the default max size is only 10000 characters. You should set it above the expected maximum size of your file. It's not recommend to set it to 0 (no limit) because something could go wrong, or you might drop in some file that shouldn't be there.

ckurtz
Path Finder

To be clear, the above stanza is in props.conf on the indexer. The inputs.conf on the forwarder would be a normal monitor stanza, such as dwaddle suggests below.

0 Karma

splk97
Engager

@gkanapathy : Thanks for sharing this. Really useful. I am facing a similar issue to ingest all line in a file as single. But the config works for me only in stand-alone environment. And not when deployed on Heavy forwarder.

Is that because logs are coming partially parsed ( and event-segmented by UF)?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...