Hacking a New SSL Certificate Structure for Ganeti 2.12.x

If you have somehow mashed the the server/client certificates on your cluster, or otherwise gotten into a deep SSL mess, and you, as we did, found the advice at GanetiAndSSL did not work for you, then maybe the following will help. We speculate that the damage to our certs was caused during the upgrade from 2.11 to 2.12. There's lots of evidence from the mailing list postings that the change in security model bit a lot of people during the upgrade process. This recipe was hacked at some cost in source diving and certificate exploring (on a working cluster) by Rob Austein, Hans Kuhn, and Randy Bush.

Be aware that this has only been tested for 2.12 and it's possible the security model will change in the future. There are no guarantees it will even work for your case and it is offered in the spirit of shared discovery.

Apologies for the examples assuming the node names of our cluster, vm[0123].

Generate Server and Client Certificates and ssconf_master_candidates_certs

Despite hints in documentation, conferences, presentations, etc.

  • There is no Certificate Authority.
  • All certificates are self-signed.
  • Client certificates are NOT signed by the server certificate.
  • Things break if the CN is not the same for all certificates.
  • The client.pem files we found were all ancient X.509v1 certificates with no extensions etc.
  • --no-ssl in /etc/default/ganeti didn't allow us to run without SSL

The following script gets a list of {nodename, uuid} and generates certificates and a new ssconf_master_candidates_certs file.

#!/bin/sh -

make_cert() {
    local filename=$1 hostname=$2 
    openssl req -batch					\
	    -newkey rsa:2048 -keyout $filename -nodes	\
	    -x509	     -out    $filename		\
	    -sha1					\
	    -days   1825				\
	    -subj   "/CN=$hostname"

make_cert server.pem

rm -f

# the following use of gnt-node assumes a semi-working cluster if not,
# you need to hack a file of the form {hostname uuid} for all hosts.
# this can be rescued from the old ssconf_master_candidates_certs
#cat nodehack |
gnt-node list -o name,uuid --no-headers |
while read hn uuid
    make_cert $fn
    fp="$(openssl x509 -noout -fingerprint -sha1 -in $fn | awk -F= '{print $2}')"
    echo >> "${uuid}=${fp}"

This generates a server.pem, a set of {nodename}.client.pem files, and a file which you will distribute to the nodes in a later step.

Updating the /var/lib/ganeti/

The hashes of the certificates need to be corrected in the /var/lib/ganeti/ file. This code needs a copy of the file in your working directory.

#!/usr/bin/env python
import json

with open("", "r") as f:
  candidate_certs = dict(line.strip().split("=") for line in f)

with open("", "rb") as f:
  config = json.load(f)

config["cluster"]["candidate_certs"] = candidate_certs

with open("", "wb") as f:
  json.dump(config, f)

And it generates a file to be pushed in the next step.

Pushing the Files to Nodes

The generated certificates need to be pushed to all nodes, along with the ssconf_master_candidates_certs file and the

Apologies that this example uses our node names vm[0123] and is run as root.


for i in 0 1 2 3; do 

   rsync vm$i.client.pem vm$i:/var/lib/ganeti/client.pem
   ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/client.pem
   ssh vm$i chmod 440 /var/lib/ganeti/client.pem
   rsync server.pem vm$i:/var/lib/ganeti
   ssh vm$i chown gnt-masterd:gnt-masterd /var/lib/ganeti/server.pem
   ssh vm$i chmod 440 /var/lib/ganeti/server.pem

   rsync \
   ssh vm$i chown root:root /var/lib/ganeti/ssconf_master_candidates_certs
   ssh vm$i chmod 444 /var/lib/ganeti/ssconf_master_candidates_certs

   rsync vm$i:/var/lib/ganeti/
   ssh vm$i chown gnt-masterd:gnt-confd /var/lib/ganeti/
   ssh vm$i chmod 640 /var/lib/ganeti/


Declare Victory

At this point, we were able to restart ganeti on all nodes. You'll need to log into each server to run this command, because the master isn't able to communicate to the nodes yet.

service ganeti restart

and successfully verify the cluster

gnt-cluster verify

We were even able to generate a new generation of credentials. This command generates a single server.pem that is distributed to all nodes, as well as a unique per-node client.pem that is distributed to the appropriate node. ssconf_master_candidates_certs is generated, and finally the ganeti service is restarted on all nodes.

gnt-cluster renew-crypto --new-cluster-certificate

Oddly, the client.pem objects, presumably generated by Ganeti itself when we told it to renew-crypto, were X.509v3 CA certificates, with BasicConstraints, AKI, and SKI extensions. Still self-signed, and no evidence that they signed anything else.

Last modified 4 years ago Last modified on Aug 3, 2016, 10:24:34 PM