Getting Data In

Forwarder: Need to send a file with same file name ( updated every 2 hours) multiple times a day

venmany
New Member

We have a strange issue wherein the file is not being forwarded using universal splunk forwarder.

For a given day, splunk forwarder is forwarding the csv input file from the monitored path 2 times only and stops sending afterwards.

The file name for a given day is unique ( for example, test.csv for 6th Oct 2017) and hence we have enabled cyclic redundancy check etc in inputs.conf.

crcSalt =
initCrcLength = 1048576

But, if I am renaming the file in OS level, its readily sent to splunk indexer.

0 Karma

mattymo
Splunk Employee
Splunk Employee

you can use a really handy command on the UF to see WHY splunk is not forwarding any given file, so that you can make decisions on input configs accurately.

Try

./splunk list inputstatus on a 6.3+ forwarder and search for your filename to see why exactly Splunk's tail reader is not linking what it is finding. Direct the output to to a file to make reading a bit easier.

Also you could search

index=_internal source=*splunkd.log TailReader to see whats up as well

This should lead you to the best config for ensuring you always read this file. You are definintely on the right track with initCRC because csvs can have long headers....but you may also just need crcSALT= although if you can avoid it with intiCRCLength then I'd go tat route...sometimes you just gotta do the salt.

- MattyMo
0 Karma

venmany
New Member

Thanks for the suggestion. I have deleted initCRCLength from input file and it works whenever the input file content undergo changes.
But immaterial the file content is changed or not, I need the file under the monitored path is forwarded to the =indexer. Should I delete crcSALT= and add initCRCLength= 1048576 in that case?

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hmm, not sure if I follow.

Splunk's file monitor logic will read a file and send its entire contents to the indexer, then monitor that file for changes (see docs for more), to ensure we then send those too.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

So are you saying you want the entire file resent when there is a change?

- MattyMo
0 Karma

gjanders
SplunkTrust
SplunkTrust

Can you post the entire inputs.conf stanza ? Or a splunk btool inputs list with the relevant info?

0 Karma

hardikJsheth
Motivator

In my opinion the program which is writing the csv file is actually creating new file with same name rather than appending results to the csv file. Can you try to ensure that the data generating program appends data to csv file instead of creating new file?

0 Karma

venmany
New Member

Hi.. The file at times might not have any change in the content. But in that scenario as well, I need forwarder to send the data to indexer.

0 Karma

mattymo
Splunk Employee
Splunk Employee

while it is worth checking that you can optimize the source of the data for splunk, you want to spend your credits wisely. asking people to change the way things are done doesn't exactly lead to path of least resistance for adoption. just 2 cents.

- MattyMo
0 Karma

hardikJsheth
Motivator

I agree to your views. I just shared this, as I had faced similar issue in my environment, hence just thought of sharing it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...