Home‎ > ‎

Installation Notes


Setting up the head node

Installing the OS

We use a base of Ubuntu Server 9.10. We should be using 64-bit (for better image compatibility), but have actually been using 32-bit. Installing any Linux on our XServe1,1 nodes is a huge pain in the ass, because they don't include a legacy bootloader like all the other non-XServe Macs do. We can fix this though:
  • RAID prevents EFI images from loading from the disk at boottime
  • Use efi-grub.tar from Christopher Smart, throw it on a USB stick, and boot off it. It will save you a lot of trouble (and you don't even need rEFIt). It allows for an extremely configurable installation, and will download the packages as needed from the network. Includes installers for Ubuntu, Debian, and Fedora.
  • Do not install the eucalyptus/cloud computing packages yet! They will break and be hard to configure and will make you sad.
  • We never did get RHEL working, even after passing innumerable flags to the kernel. It always hung in grub. 
  • You can make a videocard-less node debuggable by ensuring that it has a serial console. To do this, use a grub cfg like this:
set timeout=1
set default=0
menuentry "Ubuntu installed" {
        fakebios
        search --set -f /vmlinuz
        linux /vmlinuz video=efifb vga=normal noefi acpi=force root=/dev/sda3 console=tty0 console=ttyS0,115200n8
        initrd /initrd.img
}
  • You will also need to setup the ttyS0 console to enable serial support. In Ubuntu 9.10, you can do this by creating /etc/init/ttyS0.conf:
# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 115200 ttyS0 vt102

Networking

DHCP

We need to set up the head node to distribute IP addresses to the slave nodes. 
Relevant files to edit:
  • /etc/dhcp3/dhcpd.conf, e.g:
option domain-name "neuro.gatech.edu";
option domain-name-servers 130.207.244.244, 128.61.244.254;

default-lease-time 60000;
max-lease-time 7200;

subnet 192.168.3.0 netmask 255.255.255.0 {
range 192.168.3.50 192.168.3.100;
option routers 192.168.3.254;
}

host networkswitch {
hardware ethernet 00:0c:db:8a:52:50;
fixed-address 192.168.3.250;
}

host node015 {
        hardware ethernet 00:19:e3:e7:35:be;
        fixed-address 192.168.3.15;
}

host node016 {
        hardware ethernet 00:19:e3:e7:42:36;
        fixed-address 192.168.3.16;
}

authoritative;
  • /etc/defaults/dhcp3-server: make sure that the correct interface is specified to distribute IPs to the private subnet only (e.g. on eth0)! Make sure you don't accidentally start the DHCP server on the public network and cause havoc.

Interfaces/udev

We might think about wiping /etc/udev/rules.d/70-persistent-net.rules before copying the images to each node.
Edit 6/17/2013 by SK: Fix for new addressing.

/etc/network/interfaces:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth1
iface eth1 inet static
address 143.215.230.42
netmask 255.255.248.0
network 143.215.224.0
gateway 143.215.224.1
broadcast 143.215.231.255
dns-nameservers 8.8.8.8

auto eth0
iface eth0 inet static
address 192.168.3.254
netmask 255.255.255.0
broadcast 192.168.5.255

NAT

Network Address Translation (NAT) is necessary to allow the nodes to access the interwebs. This is important so that they can easily update themselves and sync their clocks via NTP.

To enable NAT, run these commands:
sudo iptables -P FORWARD ACCEPT
sudo iptables --table nat -A POSTROUTING -o eth1 -j MASQUERADE

To have these run automatically at boot, you can add them to /etc/rc.local.

In addition, you will need to edit /etc/sysctl.conf and uncomment the lines for ip forwarding:
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
net.ipv6.conf.all.forwarding=1

Reboot to have the changes take effect, or run:
sudo sysctl -w net.ipv4.ip_forward=1

NTP

All clocks should be sync'd to tick.gatech.edu, the campus NTP server.

APT sources

All sources will work (though some are more up to date than others). However, most sources are insanely slow... compared to the GaTech repository. URLs:
http://www.gtlib.gatech.edu/pub/ubuntu/
ftp://www.gtlib.gatech.edu/pub/ubuntu/

DNS

Since I'm too lazy to setup a DNS server, I'm going to use a poor man's server: the /etc/hosts file. This should be installed on all the slave nodes (and the head node).
Edited 6/17/2013 by SK: to reflect new IP range for BME, also correct the hostnames for each node.

127.0.0.1       localhost
143.215.230.42  brain.neuro.gatech.edu    brain
192.168.3.1     ganglion001.neuro.gatch.edu ganglion001
192.168.3.2     ganglion002.neuro.gatch.edu ganglion002
192.168.3.3     ganglion003.neuro.gatch.edu ganglion003
192.168.3.4     ganglion004.neuro.gatch.edu ganglion004
192.168.3.5     ganglion005.neuro.gatch.edu ganglion005
192.168.3.6     ganglion006.neuro.gatch.edu ganglion006
192.168.3.7     ganglion007.neuro.gatch.edu ganglion007
192.168.3.8     ganglion008.neuro.gatch.edu ganglion008
192.168.3.9     ganglion009.neuro.gatch.edu ganglion009
192.168.3.10    ganglion010.neuro.gatch.edu ganglion010
192.168.3.11    ganglion011.neuro.gatch.edu ganglion011
192.168.3.12    ganglion012.neuro.gatch.edu ganglion012
192.168.3.13    ganglion013.neuro.gatch.edu ganglion013
192.168.3.14    ganglion014.neuro.gatch.edu ganglion014
192.168.3.15    ganglion015.neuro.gatch.edu ganglion015
192.168.3.16    ganglion016.neuro.gatch.edu ganglion016

XServe RAID Space

The RAID space is formatted as two Level 5 RAIDs. They are located in /dev/sda and /dev/sdb on the head node. Using mdadm those two drives were striped to level 0 RAID to create /dev/md0, a single 3.6Tb drive. Because it uses software striped RAID, one should not consider this a suitable backup location and no guarantees are provided as to the stability of the data. /dev/md0 is mounted to /mnt/data which is shared via NFS to all IP's in the 192.168.3.0/24 subnet (no authentication necessary). The filesystem on /mnt/data is ext3.

Mounting the xserve using Snow Leopard requires using the Disk Utility to create an NFS mount point. It required adding the -P option for me to get things to work and in general the options should be: brain.neuro.gatech.edu:/mnt/data on /Volumes/brain_raid (nfs, nodev, nosuid, automounted, nobrowse)


User accounts

In order to have a nice default environment for new users, we configured the adduser skeleton directory:
Modified /etc/skel/.bashrc:

Modified /etc/skel/.profile:

Created /etc/skel/bin/ml:

Now that the XServe RAID is up and running again, new non-admin accounts will be generated with the home directory on /mnt/data (which is the mounted RAID). This will make it easier to SCP into the proper directory. MATLAB on all nodes appears to be able to traverse the directories but they must use absolute referencing ("/mnt/data/username/") and there is no protection. That is anybody can access any of your data using MATLAB.


Eucalyptus

Installing Eucalyptus

Carefully follow UEC "Package Install" instructions from Ubuntu. The only exception is when you install 

How to seed an image into the cloud

First, you need to obtain an image. In these instructions we assume that you've already obtained a KVM image. One easy way to obtain a KVM image is to use to build and download one. For example, I built an Ubuntu 9.04 KVM image with a LAMP stack and downloaded it. 
  • Once you've obtained the KVM image, you'll need to extract the kernel and initrd images if they aren't already provided for you. To do this, you'll need to mount the image by following the instructions at Dustin Kirkland's blog (Kirkland is a Canonical-paid developer that works on the Ubuntu packages for Eucalyptus). In short:
# losetup /dev/loop0 foo.img
# kpartx -av /dev/loop0
# mount /dev/mapper/loop0p1 /mnt
...
# unmount /mnt
# kpartx -dv /dev/loop0
# losetup -d /dev/loop0
  • Once mounted, the copy the kernel and initrd from the /boot directory into some place convenient.
  • Now we more or less follow the Eucalyptus Image Management instructions:
    • Assume that we have a kvm image called "ubuntu904-lamp-kvm-1416075-1267241411.img".
    • Assume that we have an initrd image called "initrd.img-2.6.28-11-server"
    • Assume that we have a kernel image called "vmlinuz-2.6.28-11-server"
    • We'll need to be bundling (processing), uploading, and registering each image separately. Start with initrd and kernel.
euca-bundle-image -i 
vmlinuz-2.6.28-11-server --kernel true

euca-upload-bundle -b kernel-bucket -m /tmp/vmlinuz-2.6.28-11-server.manifest.xml

euca-register kernel-bucket/
vmlinuz-2.6.28-11-server
vmlinuz-2.6.28-11-server.manifest.xml

    • Now y
      ou've finished the kernel image, now do the initrd image
euca-bundle-image -i initrd.img-2.6.28-11-server --ramdisk true

euca-upload-bundle -b ramdisk-bucket -m /tmp/
initrd.img-2.6.28-11-server.manifest.xml

euca-register ramdisk-bucket/
initrd.img-2.6.28-11-server.manifest.xml

    • Note that in each step above after euca-register it generated an ID. The one starting with eki is the kernel image ID, and the one starting with eri is the ramdisk image ID. Assume the kernel ID is something like eki-BD7214F9, and the ramdisk ID is
      eri-BCD817FA
      eri-BCD817FA. Now generate the machine image
euca-bundle-image -i
 ubuntu904-lamp-kvm-1416075-1267241411.img --ramdisk eri-BCD817FA --kernel eki-BD7214F9

    euca-upload-bundle -b image-bucket -m /tmp/
    ubuntu904-lamp-kvm-1416075-1267241411.manifest.xml

      euca-register image-bucket/ubuntu904-lamp-kvm-1416075-1267241411.manifest.xml



      How to run an image on the cloud

      This assumes that you've already got an image seeded into Eucalyptus. We use a KVM image obtained from ElasticServer, as explained above. Note, this only works with images that have preconfigured passwords! Images that try to dynamically pull credentials fail to do so in my experience. These stupid images include the prepackaged ones from Eucalyptus and Ubuntu store.
      • log into the web interface @ https://vm1.neuro.gatech.edu:8443/
        • default credentials:
          • user: admin
          • password: cluster2009
        • click the "Download Credentials" button, which will download a zip file
      • on your Linux box, install euca2ools (available in the repositories for Ubuntu Karmic and above), or just use vm1.neuro.gatech.edu
      • unzip -d ~/.euca euca2-admin-x509.zip 
        • (substitute the name of the credentials zip file you downloaded for euca2-admin-x509.zip). This sets up the credentials you'll need to access the cloud using euca2ools.
      • source ~/.euca/eucarc 
        • this sets up a bunch of environment variables you'll need for using euca2ools. You might want to put this line at the end of your ~/.bashrc file so you don't have to run it each time you log in.
      • euca-run-instances -t m1.large --ramdisk eri-BCD817FA emi-1DCC1847  euca-run-instances -t c1.medium emi-1DD41841
        • -t m1.large: this specifies the type of instance, which in turn specifies the amount of resources allocated to the instance. Look in the cloud web interface (in the Configuration tab) for specific numbers.
        • --ramdisk eri-BCD817FA: this specifies the ramdisk to use with the system image. The machine image actually specifies it's own default image, but apparently I didn't create it correctly. So I recreated it and this is the ramdisk that works.
        • emi-1DCC1847 emi-1DD41841: this is the machine image that provides the root file system. It also specifies a kernel (vmlinuz...) to link to and the ramdisk (initrd...) associated with it. Note that we override the associated ramdisk (see above).
      • watch -n5 euca-describe-instances
        • Now we play the waiting game. This command allows us to monitor the instance as it initializes and boots. It will start in the "pending" phase while the machine image is copied to a node and resources are allocated. It will turn into "running" when it actually starts booting. Eventually, it will obtain an IP address (to replace 0.0.0.0), and you can then access the instance.

      Useful commands

      • euca-describe-availability-zones verbose
      • euca-describe-instances
      • euca-get-console-output
      • euca_conf

      Resources

      Troubleshooting

      Eucalyptus networking hijacking

      If the Eucalyptus cluster controller (eucalyptus-cc) hijacks the networking, you can solve it by change the mode. A few special procedures need to be followed too:
      sudo stop eucalyptus CLEAN=1
      [edit /etc/eucalyptus/eucalyptus.conf and change VNET_MODE to SYSTEM]
      sudo start eucalyptus CLEAN=1
      The CLEAN=1 arguments tells eucalyptus to wipe /var/lib/eucalyptus/CC, which holds some state information about networking and configuration. It's the equivalent of /etc/init.d/eucalyptus stopclean, but since Ubuntu is converting init.d scripts to Upstart (bah!), this hack is required.

      If eucalyptus is still screwing around with networking, well, you're kind of screwed. I think I fixed it by removing all eucalyptus components and reinstalling (and rebooting and reciting magical incantations).

      Instances not starting properly

      Be sure that you've allocated enough memory to the instance. I tried to start a prepackaged Ubuntu image and noticed that there were lots of errors on the console. 

      Can't get into an instance

      Be very careful about the source of your image. The images from Eucalyptus and Ubuntu are frustratingly difficult to access once you've got it running. They ask you to generate a keypair (using euca-gen-keypair) and pass this private key as an argument to euca-run-instance. However, this requires support from the cloud controller too, because these instances do something like:
      # simple attempt to get the user ssh key using the meta-data service
      mkdir -p /root/.ssh
      echo >> /root/.ssh/authorized_keys
      curl -m 10 -s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa' >> /root/.ssh/authorized_keys
      echo "AUTHORIZED_KEYS:"
      echo "************************"
      cat /root/.ssh/authorized_keys
      echo "************************"
      That means there has to be a 169.254.169.254 server running, which doesn't always exist (it is not started in STATIC or SYSTEM mode). I filed a bug, and got this response:
      The Eucalyptus upstream will default to key injection if the metadata
      service is not available. On the UEC, key injection is turned off. The
      images on the Eucalyptus website will try the metadata service, but if
      you are running vanilla Eucalyptus, it will attempt to do key injection
      as well.

      I recommend using an image that has a real user and password. Build your own, or use one from ElasticServer.

      euca-describe-availability-zones verbose reports no available resources

      Make sure you've registered your nodes. I haven't had much success in getting errant nodes to register properly, except by uninstalling eucalyptus on everything and then reinstalling, methodically following the Ubuntu Package Install instructions

      I did get this to fix the problem:
      • delete the errant nodes from /etc/eucalyptus/eucalyptus.conf (find the NODES="x.x.x.x" line)
      • re-register using sudo euca_conf --discover-nodes

      Networking not working in MANAGED-NOVLAN mode

      Possible issues
      • DHCP server on private subnet is interfering. I noticed that a lease is being given to 169.254.169.254.
      • [008300][EUCAWARN  ] in MANAGED-NOVLAN mode, priv interface 'eth0' must be a bridge, tunneling disabled

      • [Tue Mar  2 13:53:20 2010][001452][EUCAERROR ] bad input params to vnetAttachTunnels()
      • [Tue Mar  2 13:53:20 2010][001452][EUCADEBUG ] failed to attach tunnels for vlan 10 during maintainNetworkState()
      • [Tue Mar  2 13:53:20 2010][001452][EUCAERROR ] network state maintainance failed
      More possible issues:

      The following pops up in /var/log/syslog if you attempt to start an instance using hybridfox.  I am unclear as to how to fix the DHCP, but here are some links:
      http://debianclusters.cs.uni.edu/index.php/Troubleshooting_DHCP
      http://tldp.org/HOWTO/DHCP

      Mar  2 17:55:47 vm1 dhcpd: Internet Systems Consortium DHCP Server V3.1.2
      Mar  2 17:55:47 vm1 dhcpd: Copyright 2004-2008 Internet Systems Consortium.
      Mar  2 17:55:47 vm1 dhcpd: All rights reserved.
      Mar  2 17:55:47 vm1 dhcpd: For info, please visit http://www.isc.org/sw/dhcp/
      Mar  2 17:55:47 vm1 dhcpd: WARNING: Host declarations are global.  They are not limited to the scope you declared them in.
      Mar  2 17:55:47 vm1 dhcpd: Wrote 0 deleted host decls to leases file.
      Mar  2 17:55:47 vm1 dhcpd: Wrote 0 new dynamic host decls to leases file.
      Mar  2 17:55:47 vm1 dhcpd: Wrote 0 leases to leases file.
      Mar  2 17:55:47 vm1 dhcpd: 
      Mar  2 17:55:47 vm1 dhcpd: No subnet declaration for br0 (0.0.0.0).
      Mar  2 17:55:47 vm1 dhcpd: ** Ignoring requests on br0.  If this is not what
      Mar  2 17:55:47 vm1 dhcpd:    you want, please write a subnet declaration
      Mar  2 17:55:47 vm1 dhcpd:    in your dhcpd.conf file for the network segment
      Mar  2 17:55:47 vm1 dhcpd:    to which interface br0 is attached. **
      Mar  2 17:55:47 vm1 dhcpd: 
      Mar  2 17:55:47 vm1 dhcpd: 
      Mar  2 17:55:47 vm1 dhcpd: Not configured to listen on any interfaces!

      Known issues
      • Switch does not pass VLAN packets, so MANAGED mode will not work. You must use MANAGED-NOVLAN. This was tested using vconfig.
      • Firestarter gets in the way. Don't use it.

      Outbound connections timing out on the head node

      This may be an issue with the routing tables. Run the route command to make sure that the default route is set properly - if it's set to an IP on the private subnet, you have a problem. Run route del default to remove that route, and make sure there's still a default route with an outside ip (e.g. 128.61.139.1). I'm not certain, but I think you can set the system to default to one interface in /etc/network/interfaces, by specifying a dns-nameservers line on the interface you want to route through.

      Another temporary fix: ifdown eth0, ifup eth0.


      Lusty Lynx kernel does not boot on the XServes

      Limited solution: revert to 9.10 kernel (2.6.31-19-generic).
      The full panic output is stored on this subpage.

      Tried:
      • recompiling Ubuntu Kernel from sources
      • installing upstream Debian kernels (some problem on both linux-image-2.6.31-02063112-generic and linux-image-2.6.32-02063209-generic amd64 kernels)

      Ganglion011 Hard-drive Error


      The hard drive to be used with the Ganglion011 node would not boot after being imaged from either Ganglion015 or Ganglion016.  I did a filesystem check, but things still didn't seem to work.

      Performed a test of bad-blocks on the drive: 
      sudo e2fsck -c /dev/sdb3

      Bad blocks found, but the drive still won't boot.  It gets stuck trying to load the SSH daemon.

      Another interesting thing, shutdown ganglion001 node, plugged in serial cable and tried to boot ganglion014 HD.  No boot.  Tried booting again with ganglion001 HD. No boot.  Retried.  No boot.  Retried.  No boot.  Unplugged Serial.  No boot.  Retried.  Booted.  Shutdown.  Booted.  Shutdown.  Booted.

      Serial cable may be sending some crappy signals maybe?

      Trying ganglion014 HD back in ganglion001 node with no serial cable.  No boot. Shutdown. No boot. FML.  Now try a tricky-trick.  Hold down power button until lights start flashing (clear NVRAM?), boots ganglion001 HD first time.

      Put fsck'ed ganglion011 HD in ganglion011 node, hold down power button until lights flash, presto works!
      ċ
      eucalyptus.conf.head
      (10k)
      Dustin Li,
      Mar 1, 2010, 4:45 PM
      ċ
      eucalyptus.conf.node
      (10k)
      Dustin Li,
      Mar 1, 2010, 4:45 PM
      Comments