I am not using deployment server in our environment yet. I have non-clustered search head and three indexers. If I want to create custom app where should I create it. on search head server or app server (indexer)?
I have same setup in A side (primary) and B side (standby). Both active - active.
A side - one search head and three app servers (indexers).
B side - one search head and three app servers (indexers).
I know this will consume double license but that's the requirement.
It will depend on the functionality of the app you want to create.
Example:
If it contains knowledge objects (views/dashboards, reports, alerts, etc.) it should go in the search head.
If it contains index time operations (e.g. data routing, anonymization, etc) it should go into Indexers/heavy forwarder
Check Splunk's guide to app development:
http://dev.splunk.com/view/SP-CAAAFD7#createapp
Where the app goes depends on what the app does. Apps that will contain search-time objects like dashboards, alerts, and reports go on search heads. Apps that contain index-time properties for processing events (props.conf and transforms.conf) go on indexers. Apps that define inputs should go on forwarders.
Wherever you put the app, you will have to copy it to the opposite manually, of course.