Getting Data In

Why is CSV data not getting parsed while being monitored on server with a universal forwarder?

muralianup
Communicator

We have a remote server where some CSVs are stored and the directory set to be monitored by Splunk. Now, if I upload the same CSV locally to Splunk (indexer/deployment), it seems like parsing is working fine, but the same file in the remote directory is not parsing or extracting any fields. The props & transforms on the Splunk server (indexer).

inputs.conf

[monitor:///home/test/Report/Report*.csv]
sourcetype = new_test
index = test_index
crcSalt = <SOURCE>
disabled = false

props.conf:

[ndlp_test]
TRANSFORMS-ignoreHeader = ignoreHeader
INDEXED_EXTRACTIONS = csv
SHOULD_LINEMERGE = false
DATETIME_CONFIG = CURRENT
TIME_PREFIX = \w{3}\s\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{3}\s\w{4}
MAX_TIMESTAMP_LOOKAHEAD = 8
REPORT-fields = new_test
pulldown_type = 1
KV_MODE = none
NO_BINARY_CHECK = 1

transforms:

[ndlp_test]
DELIMS = ","
FIELDS = "Field1","Timestamp","Content","Subject","Filename",

[ignoreHeader]
REGEX = ^Field1\,Timestamp\,Content,Subject
DEST_KEY = queue
FORMAT = nullQueue

Anything else I should be looking for?

lguinn2
Legend

In the inputs.conf, you specify the sourcetype as "newtest," but the sourcetype in props.conf is "ndlp_test" - props.conf does not define a sourcetype called "newtest".

In the props.conf, you specify the REPORTS field extract as "REPORT-fields = new_test", but there is no stanza named "new_test" - transforms.conf defines "ndlp_test".

You should not be using "INDEXED_EXTRACTIONS = csv" if you can extract the CSV fields at search time as you configured. Remove this line.

I don't know if these problems exist on both the local machine and the remote machine, but they would certainly cause problems. BTW, if by "remote machine" you mean a Universal Forwarder, then only the inputs.conf belongs on the remote machine.
If the remote machine is a Heavy Forwarder or Indexer, then all three files go on the remote machine.

Regardless of the type remote machine, props.conf and transforms.conf must always be on the local machine (indexer).

0 Karma

muralianup
Communicator

Oh, I am sorry, Its just I wanted to rename the sourcetype before posting. Its all same in original settings. I checked the remote machine and saw the splunk config files where under root and not accessible manually. Could it be the deployment server not able to push the configs to that machine ?

Like I said, if Im uploading the csv to the indexer manually, its parsing perfectly. Its only splunk somehow cannot do the same when the csv is on the remote machine.

0 Karma

lguinn2
Legend

Did you remove the "INDEXED_EXTRACTIONS = csv" line from props.conf?

Where did you put each of the configuration files when you were trying to collect the input from the remote machine?

0 Karma

muralianup
Communicator

Had to re-add "INDEXED_EXTRACTIONS = csv" to get it working.

0 Karma

muralianup
Communicator

Yes, INDEXED_EXTRACTIONS = csv is removed. UF was reinstalled on the remote machine. I can see the configs are pushed by the deployment server to the UF (including inputs.conf). But the same parser which works when the CSV is uploaded to the indexer doesn't seems t have any effect whent he CSVs are on the UF installed machine.

0 Karma

muralianup
Communicator

After couple of restarts and testing with new files, CSV seems to be parsing properly. Still not sure what actually did the trick. I tried the same on my test machine (Version 6.3) and that had no problems at all.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...