Thanks @thellmann  but I actually figured it out already... But since you came by this post it would be great if you'd look on this medium story I wrote about my experience in this very topic with emphasis on the lack of documentation and good examples and hopefully try to make it better for future developers My story: https://medium.com/@eylon.ronen/the-ordeal-of-using-js-inside-splunk-app-6fb3a441151d   Thank you very much ... View more

You may want to be more specific about the index in your search. It happens quite often to me that when I search index=* and than filter by source I dont get result but when i search index= source=, i get results. Probebly splunk is doing some optimizations to the search process and when you have a lot of data it doesnt want to go over all of it if not necsessery. ... View more

Wow thats a great answer. We are going to ingest about 1TB/day spread across all indexers so about 200GB/day per indexer. We are going to retain data for as long as we can. Our amount of ingested dara may grow rapidly so we will need to change retention policy. We expect a lot of concurrent searches so it is good to know that the CPUs and RAM are not wasted. We are going to have one splunk instance on each machine, but the dillema now is how to mount the disks. You said that I/O is our limit, so we need to do everything we can to make it high enough. We dont want to mount the disks all to a single LVM, because than we may hit OS I/O limit, while we hardly reach the disks limit. However it is hard for us to have several mount points and save specific indexes on each mount, thusly distributing the data across the disks, because our indexes sizes are very dynamic and we have several houndreds indexes, so managing it would be a pain. Finally we thought about having 2 mount points, one of which we will save hot/warm buckets on, and the other will contain the cold buckets. But we are not 100% sure how to spread the disks among the mounts, because we read in indexes.conf.spec that an open file handle will be saved for each warm bucket unlike cold, so its faster to search, but all over the internet people are using much more space for the cold buckets instead of warm and hot. It would really help us to know exactly the differences in all aspects between cold and warm buckets, so we know how to mount the disks, or if someone have a better solution for the disk spreading. ... View more

I didn't find any help in this page.... ... View more

well the forwarder doesnt write log when it monitors, only with batch input for some reason. Today we indexed some logs, and we saw one of the files in the forwarders log, but we could not search it... I wonder if it has something to do with the fact that we added a few indexers recently. Is there something i should update in the search head when i add indexers? ... View more

I've already looked there... Zero warnings or errors.... Also we've tried both monitor and batch input. Both had the same problem... ... View more