Inconsistent sourcetype while indexing CSV files

[erga00]

I'm indexing a CSV file and I just can't get Splunk to extract any fields or apply the proper sourcetype to the events. The desired sourcetype is acs and what I'm getting is acs-n where is some number which varies, acs-1, acs-2, etc.

I saw another answer about this issue which basically stated that it had something to do with the functionality that automatically parses field names from the header. I don't have CHECK_FOR_HEADER = true and I specify the sourcetype in the inputs.conf stanza so I don't understand what's going on.

I should mention that I have another CSV file being indexed using the same procedure as below and it works perfectly.

Has anyone run into this before?

input file:
Date,Time,Message-Type,User-Name,Group-Name,Caller-ID,NAS-Port,NAS-IP-Address
04/03/2010,00:00:00,Authen OK,c1tr1x_wds,Network Services,0004.235f.4c1c,c1tr1x_wds,10.6.70.

inputs.conf:
[monitor://\server\logs]
sourcetype = acs
_whitelist = Failed Attempts.*.csv

props.conf:
[acs]
REPORT-acs-fail = acs-fail-fields

transforms.conf:
[acs-fail-fields]
FIELDS = "date", "time", "message-type", "user", "user_group", "message", "junk1", "junk2", "src_mac", "nas-port", "src_ip"
DELIMS = ","

[erga00]

Yes. This is on a forwarder.

I thought I was setting an explicit sourcetype. Am I not understanding what that means?

[jrodman · Accepted solution]

This is the fallout from CHECK_FOR_HEADER.

There's a bug where CHECK_FOR_HEADER is applied based on filename pattern, despite not being in your sourcetype. You might be able to work around by declaring CHECK_FOR_HEADER = false explicitly in your sourcetype, or by declaring [source::.../Failed Attempts.*.csv] sourcetype=acs in props.conf, especially if you set a priority value of eg 100.

The bug is due for fixing in 4.1.4, which will be a few weeks -- we're verifying bugs as fast as we can.

[jrodman]

The expansion of the sourcetype into multiple sourcetypes allows the system to store the list of fields on a per-file basis, since there's no guarantee it doesn't change. Why the first one is -2, instead of nothing or -1, is some minor internal misbehavior that doesn't really change the situation, since you'll get multiple sourcetypes eventually. This behavior forces you to account for multiple sourcetypes in your searches manually, by eg folding them back together using eval and such. It's a pain.

[sideview]

fwiw something similar is happening to me on 4.1.6. But I do need CHECK_FOR_HEADER, so maybe I'm out of luck. Are there docs about what the "-2" suffixes are trying to do for me?

[jrodman]

In the dumbest possible case, you can either disable the [source::....csv] sourcetype=csv stanza, or turn off CHECK_FOR_HEADER for the [csv] sourcetype stanza. Obviously these approaches are a bit more intrusive.

[erga00]

Neither explicitly setting CHECK_FOR_HEADER=false or declaring [source::.../Failed Attempts.*.csv] with priority = 100 worked.

Thanks for the help. I'll open a support case and see if they can track it down.

[Lowell]

Are you doing any forwarding? I believe that CHECK_FOR_HEADER doesn't work right when forwarders are involved. (I could be wrong about that.) I've always ended up making explicit sourcetypes rather than relying on CHECK_FOR_HEADER, for whatever that's worth.