All Apps and Add-ons

Monitoring Files & Directories problem

spisiakmi
Communicator

Hi,

I created a monitor, which should check the local directory with 262 csv files. The whole set of csv files should be indexed in index1. Only 1 file has been indexed. Manual upload and indexing process into index1 of any missed csv file works. If I add any first row into missed csv file, it will be automaticaly indexed into index1. The first row of each csv file is the same (names of columns (attributes)). Can you help me, please?

0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that you must use inputs.conf and there parameter crcSalt, which define how splunk know if the file is already read or not. Unfortunately there is no way other way than get shell access to server or ask someone else add needed parameters there.

https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Howlogfilerotationishandled
--8<---

The monitoring processor picks up new files and reads the first 256 bytes of the file. The processor then hashes this data into a begin and end cyclic redundancy check (CRC), which functions as a fingerprint representing the file content. Splunk Enterprise uses this CRC to look up an entry in a database that contains all the beginning CRCs of files it has seen before. If successful, the lookup returns a few values, but the important ones are a seekAddress, meaning the number of bytes into the known file that Splunk Enterprise has already read, and a seekCRC which is a fingerprint of the data at that location.

https://answers.splunk.com/answers/35210/crcsalt-issue.html

View solution in original post

isoutamo
SplunkTrust
SplunkTrust

Hi

I think that you must use inputs.conf and there parameter crcSalt, which define how splunk know if the file is already read or not. Unfortunately there is no way other way than get shell access to server or ask someone else add needed parameters there.

https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Howlogfilerotationishandled
--8<---

The monitoring processor picks up new files and reads the first 256 bytes of the file. The processor then hashes this data into a begin and end cyclic redundancy check (CRC), which functions as a fingerprint representing the file content. Splunk Enterprise uses this CRC to look up an entry in a database that contains all the beginning CRCs of files it has seen before. If successful, the lookup returns a few values, but the important ones are a seekAddress, meaning the number of bytes into the known file that Splunk Enterprise has already read, and a seekCRC which is a fingerprint of the data at that location.

https://answers.splunk.com/answers/35210/crcsalt-issue.html

spisiakmi
Communicator

Hi soutamo,
thank you very much

It is working already. Followed steps have been made:
1. Install, Open the app Config Explorer
2. On the initial page click on the button Settings. Change write_access = false to true. Right click, choose the Save file option.
3. Open appropriate inputs.conf, find the right monitor, add this: crcSalt = at the end of monitor. Save the file.
4. Restart the server.

0 Karma

nareshinsvu
Builder

Did you try indexing using inputs.conf?

[monitor://D:\<your directory>\*.csv]
disabled = false
index = your_index
sourcetype = your_ST
interval = 10
crcSalt = <SOURCE>
0 Karma

spisiakmi
Communicator

Hi nareshinsvu,

thank you for your message. I have no access to the file system of the server. It is possible to edit the inputs.conf via gui?

0 Karma

nareshinsvu
Builder

It's equivalent to what you have done.

When you navigate to Data Inputs -> Files & Directories, what is the name of your monitor? Did you mention *.csv from your directory and did you specify your source type as csv?

0 Karma

spisiakmi
Communicator

The name of the monitor is Files & directories. There is column Full path to you data in the list of all monitors. This column is probably the name of the monitor. Right? If yes, than my monitor name is the complet local path of the detected directory. My monitor detected all files in the directory. I can see the number of detected files in the list. When I create the monitor, I choose the csv sourcetype. I have defined index, which should be relevant to this monitor. Should be set any other properties, when I create the monitor?

0 Karma

spisiakmi
Communicator

I have opened one nonindexed csv file in text editor. I added this string: crcSalt = to the header. After saving, the file has been automatically indexed in Splunk without any problem.

0 Karma

spisiakmi
Communicator

Sorry, some special chars are not allowed in this chat. crcSalt = SOURCE

0 Karma

nareshinsvu
Builder

Strange. So, Adding crcSalt to all the files resolved your issue?

Recommended approach is to do it through inputs.conf. If possible, you can request edit access to your Splunk servers' %SPLUNK_HOME%\etc\system\local directory so that you can play around with various options with conf files.

If my response helped you, kindly accept and/or upvote it.

0 Karma

spisiakmi
Communicator

Hi nareshinsvu,

it is working. Thank you for the help.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...