Build a Debian/Ganeti System and Convert libvirt Images

Initialize Hardware

Boot into MegaRaid BIOS and configure the four drives as one RAID5 for the 1RU and RAID6 for the 2RUs.

Install Debian

  • Boot Debian CD/ISO
  • Choose Install
  • Choose English, UK (so you can get UCT)
  • Choose American English
  • Name the host
  • Choose root password
  • Choose user name and password
  • Partition
    • Choose Manual Partitioning
    • Select the drive
    • Create new empty partition table
    • Select Free Space
    • Create new partition, primary, 1GB, begining, bios
    • Done
    • Select Free Space
    • Create new partition, primary, 256MB, begining, /boot, ext4, bootable
    • Done
    • Select Free Space again
    • Create a new partition
    • Accept whatever size is shown (the rest of the disk)
    • Primary, physical volume for LVM
    • Done
  • Configure LVM
    • Configure LVM accepting write changes to disks
    • Create volume group
      • Volume group name: ganeti
      • Devices for the new volume group: select only the LVM partition
    • Create Logical Volume: on ganeti, root, 16G
    • Create Logical Volume: on ganeti, swap, 16G
    • Create Logical Volume: on ganeti, var, 16G
    • Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
    • Finish partitioning and write changes
  • Finish partitioning and write changes to disk
  • Be sure it will not boot CD-ROM, and Reboot from the installed system

Finish Debian Installation

Clean up from CDROM sources

vi /etc/apt/sources.list

and delete the two CDROM entries at the top

Install homey things (it's not a computer without emacs:)

apt-get update
apt-get upgrade
apt-get install emacs23-nox
apt-get install rsync
apt-get install gcc
apt-get install bridge-utils vlan
apt-get install sudo
apt-get install unbound
usermod -G sudo -a randy

Fix hostname

echo > /etc/hostname
hostname `cat /etc/hostname`

Fix /etc/unbound/unbound.conf

        access-control: allow
        access-control: allow
        access-control: refuse
        access-control: ::1 allow
        access-control: ::ffff: allow
        access-control: 2001:deb::/48
        access-control: ::0/0 refuse

Make ISO FileSystem

lvcreate -n ISOs -L 128G ganeti
mkdir /ISOs
mkfs /dev/mapper/ganeti-ISOs

and then add it to /etc/fstab

/dev/mapper/ganeti-ISOs /ISOs   ext4    defaults        0       2

and then you can mount it

mount /ISOs

Unattended Upgrades, syslog-NG, etc.

Install Unattended Upgrading

Install syslog-ng

apt-get install syslog-ng

Hack /etc/logrotate.d/syslog-ng

*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng  2014-07-31 14:00:08.148813531 +0100
*** 29,34 ****
--- 29,36 ----
+       mailfirst
+       mail

Silence log spew to console tty by hacking /etc/syslog-ng/syslog-ng.conf

# log { source(s_src); filter(f_console); destination(d_console_all);
#                                   destination(d_xconsole); };
# log { source(s_src); filter(f_crit); destination(d_console); };

Fix Exim

dpkg-reconfigure exim4-config

Use internet mail
System name is
Only listen on local interfaces
Relay mail for noone

Debian Ganeti Specific Configuration

Edit /etc/hosts to have the real address of the host, e.g.          localhost vm0 vm1 gnt0

Fix /etc/network/interfaces

Make eth0 hang off of whatever your bridge will be called

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

auto br-lan
iface br-lan inet static
  bridge_ports    eth0
  bridge_stp      off
  bridge_fd       0
  bridge_maxwait  0

iface br-lan inet6 static
  address 2001:418:8006::4
  netmask 64
  gateway 2001:418:1::126

auto eth0.255
iface eth0.255 inet manual

auto br-svc
iface br-svc inet manual
  bridge_ports    eth0.255
  bridge_stp      off
  bridge_fd       0
  bridge_maxwait  0

auto eth1
iface eth1 inet manual

auto br-hack
iface br-hack inet static
    bridge_ports    eth1
    bridge_stp      off
    bridge_fd       0
    bridge_maxwait  0

Check /etc.resolv.comf

In theory, this looks like

                            |         this host
                  |        eth0       |
                  |                   |
                  |eth0.255   eth0.101|
                           +------> to other ganeti hosts

Install Ganeti

Set up to get Ganeti from backports on all three servers, vm0, vm1, and vm2

cat >> /etc/apt/sources.list.d/wheezy-backports.list
deb wheezy-backports main

And then install it on all three servers

apt-get update
apt-get install ganeti/wheezy-backports

Fix up drbd

echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd      # ignore any error
modprobe drbd

Initialize and Build Ganeti Cluster

gnt-cluster init \
  --master-netdev=br-internal \
  --enabled-hypervisors=kvm \
  -H kvm:kernel_path="",initrd_path="" \
  --vg-name=ganeti \
  -N link=br-cluster \
  -s \

If it barfs, try

gnt-cluster destroy --yes-do-it

then re-run gnt-cluster init ...

If you get into gnutls problems check apt-get upgrade and dist-upgrade

Check That Cluster is Happy

Fix VNC passwording

echo 'fnorgille' > /etc/ganeti/vnc-cluster-password
gnt-cluster modify -H kvm:vnc_password_file=/etc/ganeti/vnc-cluster-password

Test that the cluster was built happily

gnt-cluster verify

If you get

Fri Mar 14 05:20:55 2014   - ERROR: node volume ganeti/root is unknown
Fri Mar 14 05:20:55 2014   - ERROR: node volume ganeti/swap is unknown
Fri Mar 14 05:20:55 2014   - ERROR: node volume ganeti/var is unknown

That's OK. To make it go away

gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var

Now add vm1 and vm2 to the cluster by the following on vm0

gnt-node add -s
gnt-node add -s

which should result in gnt-node add -s
-- WARNING -- 
Performing this operation is going to replace the ssh daemon keypair
on the target machine ( with the ones of the current one
and grant full intra-cluster ssh root access to/from it

The authenticity of host ' (' can't be established.
ECDSA key fingerprint is 2f:48:91:f7:f2:dd:c4:95:da:30:d2:f4:92:6a:53:ec.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (ECDSA) to the list of known hosts.
Restarting OpenBSD Secure Shell server: sshd.
Fri Aug 22 05:37:53 2014  - INFO: Node will be a master candidate

Convert a libvirt qcow2 to a ganeti instance

get noop os type, v4 only!

echo "deb    wheezy  main" >> /etc/apt/sources.list.d/grnet.list 
wget -O -|apt-key add - 
apt-get update
apt-get install ganeti-os-noop

Find the disk size the qcow2 will require

qemu-img info <foo>.qcow2

Create the empty Ganeti instance

./do-add 32 4 <foo>

where the do-add script is


# makeVM diskGB ramGB nameFQDN


gnt-instance add \
     -t raw \
     -o noop \
     -s ${DISK}G \
     -B maxmem=${RAM}G,minmem=$((${RAM}/2))G \
     -n $NODE \
     -H kvm:vnc_bind_address= \
     --no-install \
     --no-start \
     --no-ip-check \
     --no-name-check \


Hint: Copying Images Between Systems

rsync and so forth spend a lot of time in crypto etc. If you are copying an image across a local LAN, netcat is your friend.
On the source

nc -l -p 42420 < path-to/name-of.qcow2

On the destination

nc -w3 source.fqdn 42420 > name-of.qcow2

Of course, do an md5 or sha compare of the source and destination files before using them.

Actually Convert qcow2 to a Ganeti Image

gnt-instance activate-disks <foo>

and you will get the disk ID of the instance

convert the qcow2 directly to the instance image

qemu-img convert <foo>.qcow2 /dev/ganeti/47b81077-0641-4a52-962f-393863567125.disk0

If qemu-convert wants more target disk space then you thought you had in the source disk, you will get

qemu-img: /dev/ganeti/c316d162-af4d-4ce0-8d1f-7b0b01919053.disk0: error while converting raw: No space left on device

The only solution seems to be to allocate more disk in the do-add

Now you can start the converted image

gnt-instance start <foo>

Moving Secondary Disk Drives from libvert to a Ganeti Instance

First, create the empty disk on the Ganeti image

gnt-instance modify --disk add:size=1024G <instance-name>

Then find out the magic filename of the disk

gnt-instance activate-disks <instance-name>

disk0 is the base Ganeti image disk, disk1 has been added.

The libvert secondary disk is a raw image, not a qcow. So you do a move directly into place.

On the libvert side

nc -l -p 42420 < path-to-disk.sdb

On the Ganeti side

nc -w3 vm0.iad 42420 > /dev/ganeti/f3cfa446-e214-47a6-9aca-07b5f172469f.disk1

When it is finished, you need to

gnt-instance restart <instance-name>
Last modified 5 years ago Last modified on Apr 8, 2015, 3:53:40 AM