wiki:DebianRAIDGaneti

Build a Software RAID5 Based Debian 8.0 Ganeti 2.15 Node on a UEFI BIOS

This is for UEFI based BIOS. Be sure to enable UEFI for the hard drives and for the install boot device.

Install Debian

  • Before install, it is wise to disable DHCP service so one can manuall set the system IP data
  • Also, the BIOS needs to be in UEFI mode and both the USB (or whatever) and the hard drives in UEFI
  • Boot Debian CD/ISO
  • Choose Install Debian Server
  • Choose US English or it will choose the wrong apt server set
  • Choose American English
  • CHoose Macintosh keyboard
  • Select the Ethernet interface to be used
  • Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
  • Configure Hostname & Domain Name
  • Choose user name and password
  • Let it set some stupid timezone, you can fix later
  • Partition Disks
    • Select the first drive
    • Create a new empty partition table
    • Select each of the other Drives and Create a new empty partition table
    • Select the free space on each device and
      • create a 256M ESP partition at the beginning
      • name it /boot/efi
      • Use as: EFI System Partition
      • Be sure it is bootable
      • Done setting up the partition
    • Configure Software RAID
    • Write Changes and Configure RAID
    • Create MD Device
    • Select RAID5 - I had problems with RAID10 on Ubuntu
    • Number of Devices should be all the drives you have
    • Number of Spare Drives is 0
    • Select all the free partitions (not the EFI partitions) and continue
    • Write the changes and configure RAID
    • Finish
  • Configure Logical Volume Manager
    • Keep current partition layout and configure LVM
    • Create volume group
      • Volume group name: ganeti
      • Devices for the new volume group: select /dev/md0
      • Continue
      • Keep current partioning layout and configure LVM
    • Create Logical Volume: on ganeti, root, 16G
    • Create Logical Volume: on ganeti, swap, 128G (at least RAM size)
    • Create Logical Volume: on ganeti, var, 16G
    • Finish
    • Edit the Logical Volumes to be ext4 /, swap, and ext4 /var
    • Finish partitioning and write changes

At this point, the partitioning looks like this
partitioning.jpg

  • Finish partitioning and write changes to disk
  • Debian will now install and you will do the normal mirror selection etc.
  • Only install SSH Server and, System Utilities
  • I choose no automatic updates as I ansible that in later
  • Install GRUB2, except with EFI it eems not to ask
  • Be sure it will not boot CD-ROM, and Reboot from the installed system

If Debian does NOT auto-boot (thanks Joe Clark)

  • Once Debian is installed, boot the server into the EFI Shell. This can be done by hitting F6 during the BIOS boot (you will be prompted on screen) to enter the boot list manager. Select the shell from the boot list.
  • In the shell, run the following command: bcfg boot dump
  • Look at the list for the item called "debian". Remove it using the following command: bcfg boot rm <NUM>
  • Where <NUM> is the option number of the debian option (this will typically be 1).
  • Then add back the option in order to update the boot variable. To do that, execute the following command: bcfg boot add <NEXT_NUM> fs0:\EFI\debian\grubx64.efi "UEFI : Debian " Where <NEXT_NUM> is the next available boot option number (this will typically be 5).
  • Once that is done, type reset to reboot the host.

Finish Debian Installation

Install homey things (it's not a computer without emacs:)

The first thing I do is to copy root's credentials and dot-files from a known system

Then I fix /etc/ssh/sshd_config to forbid passwords

PermitRootLogin without-password
PasswordAuthentication no

And restart the ssh daemon

service ssh restart

You can do most of this from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.

apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox rsync gcc bridge-utils vlan sudo unbound
usermod -G sudo -a randy

Get the timezone to be sensible

dpkg-reconfigure tzdata

On some systemd systems, to get unbound to start reliably, I have had to

cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service

[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535

[Install]
WantedBy=multi-user.target
EOF```

Fix hostname

echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`

Fix /etc/unbound/unbound.conf

        access-control: 127.0.0.0/8 allow
        access-control: 198.180.152.0/24 allow
        access-control: 0.0.0.0/0 refuse
        access-control: ::1 allow
        access-control: ::ffff:127.0.0.1 allow
        access-control: 2001:deb::/48
        access-control: ::0/0 refuse

And restart unbound

service unbound restart

Clean up from CDROM sources

emacs /etc/apt/sources.list

and delete the two CDROM entries at the top.

We want to manage the server so install ipmitool and smartmontools

modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool

Set up IPMI

# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
#ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 ipaddr 192.168.0.120
ipmitool lan set 1 netmask 255.255.255.0
#ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan set 1 defgw ipaddr 192.168.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list

Set up smartmontools to watch the disks

apt install smartmontools

And check that smartd is running

root      1457  0.0  0.0  25396  4512 ?        Ss   16:02   0:00 /usr/sbin/smartd -n

I configure smartd to do short test every day and long once a week

cat >> /etc/smartd.conf << EOF

/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF

System Services

It is good to have a mail system to send logs to the sucker who watches such things. I use exim4

apt install exim4 mailtools
dpkg-reconfigure exim4-config

Use internet mail
System name is vm0.sea.rg.net
Only listen on local interfaces
Relay mail for noone

In the wonderful world of systemd, here is how to get ntpd going

timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF

Unattended Upgrades, syslog-NG, etc.

I actually do these with ansib;e playbooks.

Install Unattended Upgrading

Install syslog-ng

apt-get install syslog-ng

Hack /etc/logrotate.d/syslog-ng

*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng  2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
        missingok
        notifempty
        compress
+       mailfirst
+       mail randy@psg.com
        delaycompress
        sharedscripts
        postrotate

Debian Ganeti Specific Configuration

Edit /etc/hosts to have the real address of the host, e.g.

cat > /etc/hosts << EOF
127.0.0.1       localhost
147.28.0.14     vm0.sea.rg.net vm0
147.28.0.15     vm1.sea.rg.net vm1
147.28.0.16     vm2.sea.rg.net vm2
147.28.0.100    gnt0.sea.rg.net gnt0
EOF

Fix /etc/network/interfaces

Make eth0 hang off of whatever your bridge will be called

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

auto br-lan
iface br-lan inet static
  address 198.180.150.29/24
  dns-nameservers 127.0.0.1 198.180.150.1 147.28.0.35
  dns-search psg.com rg.net rpki.net
  bridge_ports    eth0
  bridge_stp      off
  bridge_fd       0
  bridge_maxwait  0

iface br-lan inet6 static
  address 2001:418:8006::29/64
  gateway 2001:418:1::126

# for drbd
auto eth1
iface eth1 inet static
    address         10.0.0.100/24

Check /etc.resolv.comf

Install Ganeti

We'll need debian backports

cat >> /etc/apt/sources.d << EOF

# backports to get ganeti
deb http://http.debian.net/debian jessie-backports main
EOF
apt update

Check what version will be installed

apt-cache policy ganeti
ganeti:
  Installed: (none)
  Candidate: 2.12.4-1+deb8u3
  Version table:
     2.15.2-1~bpo8+1 0
        100 http://http.debian.net/debian/ jessie-backports/main amd64 Packages
     2.12.4-1+deb8u3 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        500 http://security.debian.org/ jessie/updates/main amd64 Packages

Whoops! So get explicit

apt-cache policy -t jessie-backports ganeti before to check
ganeti:
  Installed: (none)
  Candidate: 2.15.2-1~bpo8+1
  Version table:
     2.15.2-1~bpo8+1 0
        990 http://http.debian.net/debian/ jessie-backports/main amd64 Packages
     2.12.4-1+deb8u3 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        500 http://security.debian.org/ jessie/updates/main amd64 Packages

So now we can explicitly install 2.15

apt-get install -t jessie-backports ganeti

And it's a non-trivial dependency tree, to say the least

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 libcurl3
  python-bitarray python-cffi-backend python-crypto python-cryptography
  python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr
  python-ipaddress python-openssl python-paramiko python-pkg-resources
  python-psutil python-pyasn1 python-pyinotify python-pyparsing
  python-setuptools python-simplejson socat
Suggested packages:
  ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
  python-cryptography-doc python-cryptography-vectors python-enum34-doc
  python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
  python-pyinotify-doc python-setuptools-doc
Recommended packages:
  drbd-utils drbd8-utils qemu-kvm xen-linux-system-amd64
  xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
  fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 libcurl3
  python-bitarray python-cffi-backend python-crypto python-cryptography
  python-ecdsa python-enum34 python-fdsend python-idna python-ipaddr
  python-ipaddress python-openssl python-paramiko python-psutil python-pyasn1
  python-pyinotify python-pyparsing python-setuptools python-simplejson socat
The following packages will be upgraded:
  python-pkg-resources
1 upgraded, 25 newly installed, 0 to remove and 116 not upgraded.
Need to get 16.0 MB of archives.
After this operation, 127 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Fix up drbd

echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd      # ignore any error
modprobe drbd

The ganati indstall seems not to always install KVM or drbd, so do it manually

apt-get install kvm drbd-utils

And you will need the noop image type support

echo "deb http://repo.noc.grnet.gr/    wheezy  main" >> /etc/apt/sources.list.d/grnet.list 
wget -O - http://repo.noc.grnet.gr/grnet.gpg.key|apt-key add - 
apt-get update
apt-get install ganeti-os-noop

Initialize Master

So, first we're gonna need to initialize the master node

gnt-cluster init \
            --vg-name=ganeti \
	    --master-netdev=br-lan \
	    --enabled-hypervisors=kvm \
	    -N link=br-lan \
	    gnt0.sjc.arrcus.com}}}

Then verify

gnt-cluster verify

If you get errors of the form

Fri Jun  1 17:21:11 2018   - WARNING: node vm0.sjc.arrcus.com: volume ganeti/root is unknown
Fri Jun  1 17:21:11 2018   - WARNING: node vm0.sjc.arrcus.com: volume ganeti/swap is unknown
Fri Jun  1 17:21:11 2018   - WARNING: node vm0.sjc.arrcus.com: volume ganeti/var is unknown

Then

gnt-cluster modify --reserved-lvs=ganeti/root,ganeti/swap,ganeti/var
Last modified 22 months ago Last modified on Jun 5, 2018, 5:17:43 PM

Attachments (1)

Download all attachments as: .zip