space-delimited txt file import

[EricPartington]

I have a datasource that i export to a text file that I need to import into splunk. The file has a header that looks like this:

"Number" "Date" "Time" "Interface" "Origin" "Type" "Action" "Service" "Source Port" "Source" "Destination" "Protocol" "Rule" "Rule Name" "Current Rule Number" "User" "Information" "Product"

and one sample row of data

"180" "23Aug2010" "0:05:18" "eth-123" "name" "Log" "Accept" "snmp" "32913" "1.2.3.4" "name" "udp" "5" "" "7-name_PLOICY" "" "service_id: snmp" "word"

Fields of data follow after that. I would like to import and parse this data into splunk fields.

I have created a folder on the local filesystem (local to splunk) where i copy the files and have splunk watch for new files. This was set up with INPUTS.conf

[monitor:///home/applianceadmin/Desktop/Geneva]
disabled = false
followTail = 0
host = GenevaDRP
index = default
sourcetype = testcsvlog

I have been unsuccessful at setting props and transforms for this datatype. transforms.conf

[source::/home/applianceadmin/Desktop/Geneva/*]
sourcetype = testcsvlog
priority = 101

[testcsvlog_extractions]
DELIMS=" "
FIELDS="Number","Date","Time","Interface","Origin","Type","Action","Service","Source Port","Source","Destination","Protocol","Rule","Rule Name","Current Rule Number","User","Information","Product"

props.conf

[testcsvlog]
REPORT-testcsvlogextract = testcsvlog_extractions

I am sure there are several ways to skin this problem, what is the simplest? there must be a CSV import that i can use to change the delimeter.

One other problem, no matter what i set the sourcetype to be in the inputs.conf the sourcetype is set to another sourcetype i have defined? how do i change this, i suspect this is an indication that one of my settings are not correct.

[meno · Accepted solution]

Just from having a quick look at this question I would try this simple approach and check if an autogenerated props.conf has good results:

inputs.conf

[monitor:///home/applianceadmin/Desktop/Geneva/]
disbled = false
sourcetype = testcsvlog

props.conf

[testcsvlog] 
CHECK_FOR_HEADER = true

Then locate the autogenerated props.conf. (I think it's in $SPLUNK_HOME/etc/system/local). The extraction results there are generally a good base to start so you can remove CHECK_FOR_HEADER=true and continue finetuning the autogenerated props.conf in your user or application context.

[Lowell]

Is it possible that you have more than one inputs.conf entries that are pointing to your file? If you have explicitly included sourcetype = testcsvlog in your [monitor://...] stanza, as you have shown, then that should override any other sourcetype assignment.

Have you re-loaded your file after you made this settings change? Keep in mind that changing any of these settings will not change data that has already been indexed. Once stored into your index, it cannot be altered. You can change how you extract fields and things like that, but you cannot change the sourcetype. (Well, technically sourcetypes can be "renamed", but thats not quite the same thing.)

If you are just getting started with splunk, and you don't mind dumping and reindexing your data, you can save some headaches by clearing all your indexes and letting splunk reindex everything. Do some looking into the "splunk clean all" command.

[EricPartington]

I will take a closer look at my props file to see if that is an issue. I am not able to reindex my data, what I have been doing is deleting the data, deleting some content from a test log file then reimporting (using the crc_salt). I have a problem where each file that is reimported has the same sourcetype (incorrect) as when i began. how do i resent this?

[Paolo_Prigione · Accepted solution]

Is it a typo. Your [source::...] stanza should be moved from transforms.conf into props.conf file. I refer to:

[source::/home/applianceadmin/Desktop/Geneva/*]
sourcetype = testcsvlog
priority = 101

Also, riococo suggestion might be a good starting point.

Edit: another issue might reside in the DELIMS=" " configuration line. Based on this previous post, have you tried to state it as DELIMS="\s" or whathever char maps to a white space?

Can you please re-state this secondary problem?

no matter what i set the sourcetype to be in the inputs.conf the sourcetype is set to another sourcetype i have defined?

Is the file assigned an automated sourcetype or a custom one you have defined somewhere else? Might it be that you have "sourcetype=XXX" somewhere in the heading of your inputs.conf or props.conf files, thus defining some type of defaults?

[EricPartington]

THanks for the tip about the sourcetype defined elsewhere, turns out I was setting a sourcetype via a regex and that was matching content of this log file. THat has been fixed with a better regex.
I am guessing that precedence in the props.conf is important for this type of issue.

[meno · Accepted solution]

Just from having a quick look at this question I would try this simple approach and check if an autogenerated props.conf has good results:

inputs.conf

[monitor:///home/applianceadmin/Desktop/Geneva/]
disbled = false
sourcetype = testcsvlog

props.conf

[testcsvlog] 
CHECK_FOR_HEADER = true

Then locate the autogenerated props.conf. (I think it's in $SPLUNK_HOME/etc/system/local). The extraction results there are generally a good base to start so you can remove CHECK_FOR_HEADER=true and continue finetuning the autogenerated props.conf in your user or application context.