When we search (on stage) using index = <index name>
, no results come up and we get the following -
2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
[<indexer1>] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
[<indexer2>] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
What can it be?
I noticed you said (on stage). Does this mean you are searching production from staging? Do all the apps exist in both locations?
For example, you have a staging app called Stage_Search
. You are in this app context, and try to do a distributed search to production. If Stage_Search
does not exist as an app on the production indexers, it might throw a 255. Create the app on the indexers, doesn't need anything in it, just a stub is fine.
splunk create app Stage_Search
Do that on all the indexers (or use CM to push if in a cluster) and try your search again.
As far as I can recall, we ran out of disk space on the server.