trusted.pem and private.pem in auth/distServerKeys. How are they generated?

[koshyk]

As part of an automation activity, we want to connect Search Heads automatically using distsearch.conf as per the guide.
When the server starts, it automatically creates pem files in $SPLUNK_HOME/etc/auth/distServerKeys per server. And this then needs to be distributed to other peers for clustering. (trusted.pem and private.pem)

So the question is
- What's the logic of generating these pem files? can we generate them externally and copy to the auth Directory?
- Are these pem files used anywhere else in the system?

[rvany]

An older one, but...

as said the `pem`-files under `distSearchKeys` are simply RSA-Private- and corresponding RSA-Public-Key files. These can be created (at least in a Linux environment) using the following commands:

1. Create private key file:

`openssl genrsa -out private.pem 2048`

2. Create public key file from private key file:

`openssl rsa -in private.pem -pubout -out trusted.pem`

The `openssl` binary is located in Splunk's `bin` folder.

If you want to create a trusted connection from e.g. a Cluster Master (CM) to an Indexer (IX) manually you just do the following:

1. create a new directory under `distServerKeys` named according to your CM-hostname

2. copy the `trusted.pem` file (i.e. the public key) from the CM's `distServerKeys` directory to the newly created directory on your IX

3. restart Splunk on both (in a cluster according to restart policy/rules)

Under "Settings | Distributed search" on the CM you could then see if the connection between CM and IX is established: State = Up, Health status = Healthy

If one of the above steps failed you get: State = Down, Health status = Sick

(at least this was the output during my experiment I just ran)

[soumyasaha25]

private.pem is the private key and trusted.pem is the public key. These keys are generated the first time you run splunk. If you are adding the search peers via the cli or by editing the distsearch.conf file you will need to install the public key from the head onto your peers if you add via the ui you can skip this step by providing user credentials on the search peer and splunk will install the key for you. trusted.pem is used to establish trust between the Search heads and the peers(indexers). They are only
need to be on the indexers.
trusted.pem being the public key it should be safe to distribute it via a configuration management system (e.g. Puppet

Refer the distributed search manual for more information
also you can refer this blog for some more information on this.

[koshyk]

hi, I understand it is generated by splunk on start etc. I was looking for how it can be generated externally outside Splunk for automation purposes. (Not while starting splunk), so it can be reused and centrally stored. We are looking for creating these artefacts in an automated key-gen facility and copy to relevant servers rather than waiting for server to create itself.

[Adam2Marsh]

Hi Koshyk,

Did you ever give this ago and get it working? I want to do the same as you but can't find any documentation on whether it works or not.

Thanks,
Adam

[koshyk]

i made this work by manually copying it. That's not ideal, as I was looking for the logic of how Splunk generates it