Why are only some (91 out of 1300) JSON files from the same folder location being indexed?

[jrodsecurity]

Splunk Version 6.1.2 and Splunk 6.2.0
I have created a Data Inputs folder with roughly 1300 small JSON files in it. When Splunk Indexes the data it only grabs 91 of the files to index then no other files get indexed
I have created the proper Index for this with the proper Data Input.
I put the folder with the files locally on the same Splunk Instance to take the Forwarder out of the equation. At this point im at a bit of a loss as to why some not all the files get indexed.

Path:/Applications/Splunk/outgoing

sourcetype: _json
index: default

Number of Files: 1377

[david_rundle_fi]

I've also just encountered this same issue - 838 JSON files in target dir, less than 16 MB on disk, but only 108 indexed. Also happens from a forwarder to search head.

[emiller42]

Is this just a group of files that need to be indexed once, or are they being updated and must be tailed? If its the former and you're using typical 'monitor' stanzas, you might be running into some bottlenecks with the number of open file handles.

If this is just historical data you need to pull in, you might want to look into using a 'batch' input. (details in inputs.conf) This reads the file once, then deletes it, instead of constantly watching it for updates.

I've seen similar behavior when attempting to index IIS servers which have no log retention policies, and Splunk tries to watch multiple years worth of iis log files.