Event Count keeps increasing when monitoring CSV file

[tskarthic]

I have configured a CSV file path using Monitor files and directories option in the Add Data feature. That CSV file having 1,20,742 records(events). But when doing search in splunk, this event count is keep on increasing. I have inserted 6 records into that csv file. those records have been displayed in the splunk search. But the problem is event count. Now it shows 8,45,934 events. How is it possible since the source file having only 1,20,748 records and why the event count is keep on increasing.

Even after removing all the pipes(|) from the query, its showing the 8,45,934 only. How to avoid this problem?

[woodcock]

My suspicion is that you are replacing the entire file, not adding to it with something like echo "This is a test" >> MyLogFile. Try a proper test using something that actually adds to the bottom of the file instead of something that replaces the entire file with the same stuff plus some other stuff. It is your test methodology that is broken, not the file or Splunk.

[richgalloway]

Please share the inputs.conf stanza for that file.

[tskarthic]

inputs.conf file below:

[tcp://443]
connection_host = dns
index = main
sourcetype = syslog
[WinHostMon://MyMachine]
index = main
interval = 1800
type = Roles;NetworkAdapter;Service;OperatingSystem;Driver;Processor;Disk;Computer;Process
[monitor://C:...\Documents\Talend\APM\OSH_Data\out-apmts_aug31st.csv]
disabled = false
index = mnd_osh
sourcetype = osh_ts_csv

FYI: i have not updated this file when configure monitoring file. I just used the UI option to configure these settings and opted the "Continously Monitor" option.

[jkat54]

Please provide a sample of the csv data and your props.conf as well. I believe your line breaking is off.

[tskarthic]

pls find below the sample records in the csv file:

123456.ABC,2017-09-01T00:00:00.000Z,1,2
123457.ABC,2017-09-05T00:00:00.000Z,2,2
123458.ABC,2017-08-01T00:00:00.000Z,0,3
123459.ABC,2017-08-01T00:05:00.000Z,0,3
123460.ABC,2017-08-01T00:10:00.000Z,0,3
123461.ABC,2017-08-01T00:15:00.000Z,0,3
123462.ABC,2017-08-01T00:20:00.000Z,0,3
123463.ABC,2017-08-01T00:25:00.000Z,0,3

props.conf file:

[osh_ts_csv]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
FIELD_NAMES = resource_tag, timestamp, value, quality
TIMESTAMP_FIELDS = timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N

[richgalloway]

TIME_FORMAT should be %Y-%m-%dT%H:%M:%S.%3N%Z
The other settings look OK to me.

[tskarthic]

TIME_FORMAT given as you mentioned %Y-%m-%dT%H:%M:%S.%3N%Z.
Able to do search and getting results. the only problem is EventCount is keep on increasing.
EventCount should always equal to the records/lines in the source file. But it increased 7 times.