Getting Data In

Ignoring any data record that begins with a "#" character

teward001
Path Finder

I've got data being imported from a CSV file into a custom data type, but it's reading the first line (which begins with # in all the data sources) as an actual record.

Is it possible to get Splunk 6.0.x to ignore any line beginning with a # character?

Tags (2)
0 Karma
1 Solution

teward001
Path Finder

I figured this out, what was going on is it wasn't applying the CSV extractions, so header lines were getting included in event data.

Problem solved by switching the datatypes to all csv, but making one custom CSV with the necessary timestamp search modifications. ^.^

View solution in original post

0 Karma

nithish
Loves-to-Learn

I had similar issue and below props and transforms worked for me in my scenario where it ignored the lines started with #:

transforms.conf:


[setnull]
REGEX=^[#\r\n]
DEST_KEY=queue
FORMAT=nullQueue

props.conf:

[sourcetype]
TRANSFORMS-ignore_comments = setnull

0 Karma

teward001
Path Finder

I figured this out, what was going on is it wasn't applying the CSV extractions, so header lines were getting included in event data.

Problem solved by switching the datatypes to all csv, but making one custom CSV with the necessary timestamp search modifications. ^.^

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I've wondered the same thing and have an idea, but haven't had a chance to try it. Set your transforms.conf file to send lines beginning with '#' to nullQueue.

props.conf

[<sourcetype>]
 SHOULD_LINEMERGE = false
 TRANSFORMS-set = setnull,setparsing

transforms.conf

[setnull]
 REGEX = ^#.
 DEST_KEY = queue
 FORMAT = nullQueue

[setparsing]
 REGEX = logit
 DEST_KEY = queue
 FORMAT = indexQueue
---
If this reply helps you, Karma would be appreciated.

splunkranger
Path Finder

What he said ^

0 Karma

splunkranger
Path Finder

Can you post an example? Do you want to not index those records, or just not have them show up in search?

0 Karma

teward001
Path Finder

Try not posting comments as answers, it irks me.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...