While developing apps and testing changes I can see that resource sharing could become a problem hogging the production indexer resources, but are there any benefits to having a separate DEV server or VM? I guess are there any good arguments to justify the expense?
thanks
Reasons to have a separate DEV environment:
1) Developers can't accidentally break Prod.
2) Dev doesn't steal resources from Prod.
3) Dev can test the latest rev of Splunk without affecting Prod.
4) Simpler management. No complicated permissions on Prod for the Dev team. Just give Dev admin rights on their own box so they can shoot themselves in the foot if they want.
5) No need for Dev to wait for the Prod admin to install updates, etc.
Dev should still have read-only access to Prod, however.
1) The load of DEV searches will not affect PROD indexers, giving better performance.
1.1) Any really bad search (which people often write during development) or realtime search may kill or make an indexer terribly slow.
2) Prevents accidental delete of PROD data from DEV SH.