Getting Data In

monitor multiple files in folders

ketan_chanana
Engager

Hi

I want to monitor multiple csv files in a folder name Fwd Test on E drive. I have added below code to my inputs.conf file but Splunk is not picking up data. Do i need to make any changes in props.conf as well?

[monitor://E:\Fwd Test]
index=dir_test
sourcetype=csv
  1. My CSV files in this folder get replaced every 1 hour from Autosys jobs. With this Splunk is duplicating the data indexing. For example: if initially my file report_output has 10 rows initially and after one hour when autosys jobs replace that file with new data for total 20 rows, splunk will index 10+10+20 events. Kindly help
0 Karma

yannK
Splunk Employee
Splunk Employee

"Splunk is not picking up data." do you mean that the first version of the file was indexed, but not the next ones ?

search in the _internal splunkd.log for the file name, they may be skipped because considered as duplicates.
Please double check that your new files do not have identical header than the previous one.

and read this documents with care : http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/HowLogFileRotationIsHandled

0 Karma

linu1988
Champion

I think the data is now picked from the files but, when we update the existing csv files splunk doesn't take only the updated rows it take the whole CSV content again.. This can be reproduced easily.

0 Karma

jrodman
Splunk Employee
Splunk Employee

If your application is adding rows to the file by adding bytes to the end, then it is behaving like a logfile, and splunk will handle that.

if your application is adding rows to the file by rewriting the entire file, and the bytes are different, then it is not behaving like a logfile, and splunk is not designed to handle that.

0 Karma

srioux
Communicator

Did you restart Splunk after updating the inputs config?

0 Karma

jrodman
Splunk Employee
Splunk Employee

This question is unclear. You say that Splunk is not picking up data. Then you say that splunk is duplicating the data.
These two statements seem to be in conflict. Can you clarify?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...