I have a csv file that's giving me a headache while trying to index it.
It has 100+ columns, several of which are making life difficult by containing large amounts of things like quotes and newlines.
field1,field2,field3,field4,field5,field6
"55634","Barney","","this field behaves well","","1436504081000"
"","Fred","","Here, have some data
that will make your life very difficult
""should"" you try to parse this puppy","F6E25B","1435307738000"
(The quotes around should are intentional, there's sections of the data that look exactly like that)
I've tried using the following props, to no avail - Barney does the right thing, but Fred's line breaking goes wrong. Can someone point out where I'm going wrong?
BREAK_ONLY_AFTER=\"$
HEADER_FIELD_LINE_NUMBER=1
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=field6
This file is created completely new at a regular interval - it's a scheduled database dump. I want to index the entirety each time.
I want to keep the inputs.conf as simple as possible, only defining host, sourcetype and destination index. A parsing app on the indexer will have the props.conf.
Thanks in advance for any help.
Ok, to me the only thing that seems to be consistent in your two examples is every event ends with time (epoch). I would do is use MUST_BREAK_AFTER instead of BREAK_ONLY_AFTER.
[yourSourcetype]
MUST_BREAK_AFTER=,"\d{13}"
HEADER_FIELD_LINE_NUMBER=1
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIMESTAMP_FIELDS=field6
Sorry, I think I've given you the wrong idea with my fictional data. The actual data's last column may or may not have a value in it. I'll edit my example data when I can.