wiki:XenialInstall

Build a Software RAID Based Ubuntu 16.04 Ganeti 2.15 Node

Install Ubuntu

  • Before install, it is wise to disable DHCP service so one can manually set the system IP data
  • Boot Ubuntu CD/ISO
  • Choose Install Ubuntu Server
  • Choose US English or it will choose the wrong apt server set
  • Choose American English
  • CHoose Macintosh keyboard
  • Select the Ethernet interface to be used, I had connected eno1
  • Configure Ethernet IP Address, Netmask, Gateway, and DNS Service
  • Configure Hostname
  • Choose user name and password
  • Do not encrypt home directory
  • Let it set some stupid timezone, you can fix later
  • Partition Disks
    • Choose Manual Partitioning
    • Select the first drive
    • Create a new empty partition table
    • Select each of the other Drives and Create a new empty partition table
    • Select the free space on each device and create a 256M ESP partition named /boot/efi
    • Configure Software RAID
    • Write Changes and Configure RAID
    • Create MD Device
    • Select RAID10 - with lots of disk, I chose RAID10 for speed
    • Number of Devices should be all the drives you have, but an even number
    • Number of Spare Drives is 0, unless you had an odd number of drives
    • Select all the drives (not the EFI partitions) and continue
    • Write the changes and configure RAID
    • Finish
  • Configure LVM
    • Configure LVM accepting write changes to disks
    • Create volume group
      • Volume group name: ganeti
      • Devices for the new volume group: select /dev/md0
      • Keep partioning and write
    • Create Logical Volume: on ganeti, root, 16G
    • Create Logical Volume: on ganeti, swap, size of RAM
    • Create Logical Volume: on ganeti, var, 16G
    • Finish
    • Edit the Logical Volumes to be ext4 /, swap, and ext4 /var

At this point, the partitioning looks like this
raid10-part.jpg

  • Finish partitioning and write changes
  • Finish partitioning and write changes to disk
  • Ubuntu will now install and you will do the normal mirror selection etc.
  • I choose no automatic updates as I ansible that in later
  • Only install SSH Server, System Utilities
  • Install GRUB2, except with SuperMicro? EFI boot enabled it eems not to ask
  • Be sure it will not boot CD-ROM, and Reboot from the installed system

Finish Ubuntu Installation

Install homey things (it's not a computer without emacs:)

apt update
apt upgrade
apt dist-upgrade
apt install emacs-nox gcc bridge-utils vlan unbound
usermod -G sudo -a randy

As apt dist-upgrade probably installed a new kernel and other goodies, now is a good time to reboot.

The next thing I do is to copy root's credentials and dot-files from a known system

Fix /etc/ssh/sshd_config to forbid passwords

PermitRootLogin without-password
PasswordAuthentication no

And restart the ssh daemon

service ssh restart

You can do most of the rest from the KVM/VGA or from an ssh with password. Note that IPMI-over-LAN will have a different IP address for the LAN than the IPMI.

Set the timezone

dpkg-reconfigure tzdata

In the wonderful world of systemd, here is how to get ntpd going

timedatectl set-ntp true
cat > /etc/systemd/timesyncd.conf << EOF
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=ntp.psg.com 0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org
FallbackNTP=ntp.ubuntu.com
EOF

On some Xenial systems, to get unbound to start reliably, I have had to

cat > /etc/systemd/user/unbound.service << EOF
[Unit]
Description=Unbound is a validating, recursive, and caching DNS resolver.
After=network.target networking.service

[Service]
Type=simple
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/unbound/root.key
ExecStartPre=/usr/local/sbin/unbound-checkconf
ExecStart=/usr/local/sbin/unbound -d
LimitNOFILE=102400
LimitNPROC=65535

[Install]
WantedBy=multi-user.target
EOF

Fix hostname

echo vm0.sea.rg.net > /etc/hostname
hostname `cat /etc/hostname`

Fix /etc/unbound/unbound.conf

        access-control: 127.0.0.0/8 allow
        access-control: 198.180.152.0/24 allow
        access-control: 0.0.0.0/0 refuse
        access-control: ::1 allow
        access-control: ::ffff:127.0.0.1 allow
        access-control: 2001:deb::/48
        access-control: ::0/0 refuse

And restart unbound

service unbound restart

I hack grub to pause, havd a serial console, and let ethernet interfaces be called ethN. Edit /etc/default/grub to have

GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=6
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,9600n8"

And reconfigure grub

# update-grub

Clean up from CDROM sources

emacs /etc/apt/sources.list

and delete the two CDROM entries at the top.

We want to manage the server so install ipmitool and smartmontools

modprobe ipmi_si
modprobe ipmi_devintf
echo ipmi_si >> /etc/modules
echo ipmi_devintf >> /etc/modules
apt install ipmitool

Set up IPMI

# turn off gratuitous arp
ipmitool lan set 1 arp generate off
# ip addressing
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 147.28.0.240
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 147.28.0.1
ipmitool lan print 1
# user(s)
ipmitool user set name 2 <whatever>
ipmitool user set password 2 <secret>
ipmitool user priv 2 4
ipmitool user enable 2
ipmitool user list

Set up smartmontools to watch the disks

apt install smartmontools

And check that smartd is running

root      1457  0.0  0.0  25396  4512 ?        Ss   16:02   0:00 /usr/sbin/smartd -n

I configure smartd to do short test every day and long once a week

cat >> /etc/smartd.conf << EOF

/dev/sda -o on -S on -s (S/../.././02|L/../../3/03)
/dev/sdb -o on -S on -s (S/../.././04|L/../../4/03)
/dev/sdc -o on -S on -s (S/../.././06|L/../../5/03)
/dev/sdd -o on -S on -s (S/../.././08|L/../../6/03)
EOF

System Services

It is good to have a mail system to send logs to the sucker who watches such things. I use exim4

apt install exim4 mailtools
dpkg-reconfigure exim4-config

Use internet mail
System name is vm0.sea.rg.net
Only listen on local interfaces
Relay mail for noone

Unattended Upgrades, syslog-NG, etc.

Install Unattended Upgrading

Install syslog-ng

apt-get install syslog-ng

Hack /etc/logrotate.d/syslog-ng

*** /etc/logrotate.d/syslog-ng~ 2013-03-20 17:30:26.000000000 +0000
--- /etc/logrotate.d/syslog-ng  2014-07-31 14:00:08.148813531 +0100
***************
*** 29,34 ****
--- 29,36 ----
        missingok
        notifempty
        compress
+       mailfirst
+       mail randy@psg.com
        delaycompress
        sharedscripts
        postrotate

Ubuntu Ganeti Specific Configuration

Edit /etc/hosts to have the real address of the host, e.g.

cat > /etc/hosts << EOF
127.0.0.1       localhost
147.28.0.14     vm0.sea.rg.net vm0
147.28.0.15     vm1.sea.rg.net vm1
147.28.0.16     vm2.sea.rg.net vm2
147.28.0.100    gnt0.sea.rg.net gnt0
EOF

Install Ganeti

In Ubuntu 16.04, ganeti is in the package system, no need for hacks.

Check what version will be installed

apt-cache policy ganeti
ganeti:
  Installed: (none)
  Candidate: 2.15.2-3
  Version table:
     2.15.2-3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

So now we can explicitly install 2.15

apt install ganeti

And it's a non-trivial dependency tree, to say the least

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fping ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15 iputils-arping
  libcurl3 python-bitarray python-cffi-backend python-crypto
  python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
  python-ipaddr python-ipaddress python-openssl python-paramiko
  python-pkg-resources python-psutil python-pyasn1 python-pycurl
  python-pyinotify python-pyparsing python-simplejson python-six socat
Suggested packages:
  ganeti-doc blktap-dkms molly-guard python-crypto-dbg python-crypto-doc
  python-cryptography-doc python-cryptography-vectors python-enum34-doc
  python-openssl-doc python-openssl-dbg python-setuptools python-psutil-doc
  doc-base libcurl4-gnutls-dev python-pycurl-dbg python-pycurl-doc
  python-pyinotify-doc
Recommended packages:
  drbd-utils | drbd8-utils qemu-kvm | xen-linux-system-amd64
  | xen-linux-system-686-pae ganeti-instance-debootstrap ndisc6 libjs-jquery
The following NEW packages will be installed:
  fping ganeti ganeti-2.15 ganeti-haskell-2.15 ganeti-htools-2.15
  iputils-arping libcurl3 python-bitarray python-cffi-backend python-crypto
  python-cryptography python-ecdsa python-enum34 python-fdsend python-idna
  python-ipaddr python-ipaddress python-openssl python-paramiko
  python-pkg-resources python-psutil python-pyasn1 python-pycurl
  python-pyinotify python-pyparsing python-simplejson python-six socat
0 upgraded, 28 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.3 MB of archives.
After this operation, 141 MB of additional disk space will be used.
Do you want to continue? [Y/n] 

Fix up drbd

echo "options drbd minor_count=128 usermode_helper=/bin/true" > /etc/modprobe.d/drbd.conf
rmmod drbd      # ignore any error
modprobe drbd
Last modified 3 years ago Last modified on Oct 11, 2016, 10:38:37 PM