Splunk Dev

Does splunk have an option to ignore double quote in add data for csv file?

tamduong16
Contributor

Some of the information in my csv file have random double quote inside and Splunk seem to append the next line of data whenever it sees the quote. Therefore, it makes the event summary not accurate. When I change the quote character to single quote, it seems to do the trick but I end up with data that have double quote wrap around everything and it makes it hard to do a search.
The data in line failed attempt contain random double quote, which is why splunk try to read in the next line after the quote.
alt text

could anyone show me how do i tell splunk to ignore double quote when inputting data?

0 Karma

bsaeed_splunk
Splunk Employee
Splunk Employee

Tam,
You can use RegEx to edit the data before you index your file into Splunk. As data passes through our pipeline, we have the option to use RegEx to identify/classify the data as we write it into our Flat File Time-series Index. The answer to your question posted by sbbadri will work for you; you can these RegEx expressions that he listed to remove the double quotes " from your raw events. If you want to verify if an expression will return an expected string, use regex101.com.

"Props.conf" file can be found in the /Splunk/etc/system/default directory. It includes settings for the processing properties of your data e.g. line-breaking, character encoding, timestamp recognition, event segmentation, automated host, sourcetype matching overrides, search-time field extraction definitions. More information on props.conf stanzas: https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Propsconf

0 Karma

sbbadri
Motivator

In Indexer,

props.conf

[your sourcetype]
SEDCMD-removeDoubleQuotes= s/\"//g
or
[your sourcetype]
SEDCMD-removeDoubleQuotes= s/"//g
or
[your sourcetype]
SEDCMD-removeDoubleQuotes= s/(")//g

0 Karma

tamduong16
Contributor

Hi I'm very new to splunk. I have not touch anything else other than the search & reporting function. Could you please tell me how I could get to that file. Thanks!

Also I have to make splunk ignore " before the data is completely added. If I don't do this, the data from the next line will get append to the previous line which mess up my event summary count.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...