Universal Forwarder and CSV (from Remote System)

[nikhilmehra79]

Hi,

I have a Universal Forwarder whose source file is reading all files in a specific directory , the dir has many files including HTM and CSV files, forwarder is reading all HTML files but not CSV by default. SO indexer is not seeing that data which in CSV file....what am i missing here.

From remote system
Here is my input.conf

[default]
host = ABCD

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[monitor://D:\web\System_Availablity_Analytics\*.csv]

[nikhilmehra79]

Also where is best place to put the same props.conf in Universal Forwarder

[lukejadamec]

The best place to put global props.conf stuff is on the indexer in etc/system/local.

[lukejadamec]

Good news. If you can provide specifics then perhaps we can help.

[nikhilmehra79]

and i see the data , here is what i did
Added stanza to etc/apps/learned/local/props.conf and modified input.conf to listen
[monitor://D:\web\System_Availablity_Analytics*.csv] sourcetype = csv-2

Thanks for all help on this, lastly - i see the file being read by indexer as breaking events not the way i like - in order for me to make it learn how it reads:

In past i was building props.conf of a particular application to write logic to split where i want to, is it best to put the Global Props.conf in Search App and any new App i build will leverage that?

[lukejadamec · Accepted solution]

The inputs.conf is probably not necessary, because you already have one on the forwarder. What you will need to do is add the new sourcetype to the input stanza on the forwarder like this:

[monitor://D:\web\System_Availablity_Analytics\*.csv]
sourcetype = csv-2

[nikhilmehra79]

i dont see anything in input.conf which has changed today (search app is default), props.conf has this entry
[csv-2]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-1
SHOULD_LINEMERGE = False
pulldown_type = true

So should go to same place in Universal Forwarder?

[lukejadamec]

On the indexer, check etc/apps/?/local/inputs.conf for the input. The app might be will be what ever app you were in before you went to the manager.
The props.conf will be in the etc/apps/learned/local/props.conf
You will only need the stanzas for the input you created in inputs.conf, and the sourcetype that was created in props.conf. They should both be at the bottom.

[nikhilmehra79]

ok confused a little bit...
I put this in indexer and used GUI to map CSV file - i see it indexed the CSV file locally and it shows content on the Summary Tab

Now when i go to find indexer.conf and props.conf - i see many of them on system but not sure where it made change...which file will it write to and what i need to change before i move to unversal forwarder. Thanks for your help on this.

[lukejadamec]

The source will be different, so yes. Once you are done creating the configs and move them to the forwarder, then you can delete the input from the indexer. Don't forget to use a test index, and change the index in inputs.conf to 'main' when you move the configs to the forwarder.

[nikhilmehra79]

so if i create this through indexer ....next time this file is overwritten will it work?

[nikhilmehra79]

no its whole file...

[lukejadamec]

Is that the whole thing or one event?
I would copy one of the files to the indexer c:temp directory, and use the monitor files and directories GUI to create the inputs.conf and props.conf configs, and then transfer those configs to the forwarder.
If you do this, when you transfer the configs you will have to correct the monitor path when you move the configs to the forwarder.

[nikhilmehra79]

no it should be ASCII....

Here is CSV file

"","Availability"
"","%"
"","All"
"",""
"Element",""
"ABCD",100.00000000

"Auto Range: Previous Hour","Subject: REPORT","Created: 03/06/2014 12:55:03 PM"
"From: 03/06/2014 11:55 AM","","Time Zone: (GMT-06:00) Central Time"
"To: 03/06/2014 12:55 PM"

[lukejadamec]

Only the first one relates to your csv problem.

Does the file use a character set other than utf-8 or ascii?

[nikhilmehra79]

03-06-2014 13:14:55.651 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkForwarder\bin\splunk-netmon.exe"" splunk-netmon - Splunk network monitor is not available on this version of Windows.
03-06-2014 13:15:00.651 -0600 INFO ExecProcessor - message from ""C:\Program Files\SplunkForwarder\bin\splunk-regmon.exe"" splunk-regmon - SysmonMigrator::read - 'sysmon.conf' was not found, no migration is required.

[nikhilmehra79]

Yes i see them
03-06-2014 13:14:45.760 -0600 ERROR TailingProcessor - Ignoring path="D:\web\System_Availablity_Analytic\Report.csv" due to: Cannot checksum file due to unknown charset="AUTO".f
03-06-2014 13:14:50.635 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkForwarder\bin\splunk-MonitorNoHandle.exe"" wmain: Operating system major version 5, detected -- A minimum of 6 (VISTA/Server 2008) is required. Exitting.

[lukejadamec]

Are you seeing any errors in the splunkd log?