Ok, this is going to seem kinda lame, but I'm going to answer my own question. With the help of ^Brian^ and amrit|wrk on EFNet's #splunk channel, I was able to come up with a solution. First off, here's the first few lines of one of the files in question:
Date, Time, Switch Name, Group Name, Key/Info Field, Reg1 Name, Reg1 Value, Reg2 Name, Reg2 Value, Reg3 Name, Reg3 Value, Reg4 Name, Reg4 Value, Reg5 Name, Reg5 Value, Reg6 Name, Reg6 Value, Reg7 Name, Reg7 Value, Reg8 Name, Reg8 Value, Reg9 Name, Reg9 Value, Reg10 Name, Reg10 Value, Reg11 Name, Reg11 Value, Reg12 Name, Reg12 Value, Reg13 Name, Reg13 Value, Reg14 Name, Reg14 Value, Reg15 Name, Reg15 Value, Reg16 Name, Reg16 Value, Reg17 Name, Reg17 Value, Reg18 Name, Reg18 Value, Reg19 Name, Reg19 Value, Reg20 Name, Reg20 Value, Reg21 Name, Reg21 Value, Reg22 Name, Reg22 Value, Reg23 Name, Reg23 Value, Reg24 Name, Reg24 Value, Reg25 Name, Reg25 Value, Reg26 Name, Reg26 Value, Reg27 Name, Reg27 Value, Reg28 Name, Reg28 Value, Reg29 Name, Reg29 Value, Reg30 Name, Reg30 Value, Reg31 Name, Reg31 Value, Reg32 Name, Reg32 Value
01-30-2014,06:30:07,Host_Name,NPILNS,0,LMTRU,2562,NTERMATT,550,ORIGABN,1768,PERCLFL,0,ORIGFAIL,111,ORIGBLK,0,TERMBLK,0,NORIGATT,6334
01-30-2014,06:30:07,Host_Name,MS,MS 0,MSERR,0,MSFLT,0,MSDIA,0,MSDIAF,0,MSMBP,0,MSMBU,0,MSSBU,0,MSCDERR,0,MSCDFLT,0,MSCDDIA,0,MSCDDIAF,0,MSCDMBP,0,MSCDMBU,0,MSCDSBU,0,MSPTERR,0,MSPTFLT,0,MSPTDIA,60,MSPTDIAF,0,MSPTMBP,0,MSPTMBU,0,MSPTSBU,0
01-30-2014,06:30:07,Host_Name,MS,MS 1,MSERR,0,MSFLT,0,MSDIA,0,MSDIAF,0,MSMBP,0,MSMBU,0,MSSBU,0,MSCDERR,0,MSCDFLT,0,MSCDDIA,0,MSCDDIAF,0,MSCDMBP,0,MSCDMBU,0,MSCDSBU,0,MSPTERR,0,MSPTFLT,0,MSPTDIA,60,MSPTDIAF,0,MSPTMBP,0,MSPTMBU,0,MSPTSBU,0
Here is my new monitor:// configuration that works:
[monitor:///home/users/c20dd/[A-Z]*/omdd/*.CSV]
host_regex = \w+\.\w{10}\.\d{4}\.(\w+).*
index = for_testing
sourcetype = C20_OM
initCrcLength = 4096
disabled = 0
The line that fixed everything was the initCrcLength = 4096. The problem was that my CSV header was so long that for some reason (I don't have a good answer on this), even with the crcSalt specified before, Splunk wasn't able to uniquely identify the contents of my files. With the increased CrcLength, the files were gobbled up perfectly.
amrit was the guy that developed the new field extraction code introduced in Splunk 6.0, and even he wasn't able to explain why Splunk wasn't able to handle my data without any Crc configuration present. He seemed to think that his new code should handle it. Oh well, at least it works now 🙂
... View more
