Equipment/Landin

Landin
Model	Xyratex HS-1235T (2U Version with 12 3.5" SAS Bays)
Sub-category	Systems
Status	Good working order
Training requirement	yes
Training link	Unknown
ACnode	no
Owner	LHS
Origin	Donation
Location	Basement rack
Maintainers	Sysadmin team

Multi-use server server for services in Ujima House

The system was named after the British computer scientist Peter Landin who was instrumental in using lambda calculus to model a programming language, leading to functional programming.

Please do not install anything directly on Landin (Make a VM)

Info

• IP: 10.20.20.10
• DNS: landin.london.hackspace.org.uk
• Access: LDAP

Stats

Landin is a Xyratex HS-1235T (OEM platform for IBM XIV, Dell Compellent, LaCie 12Big, Pure FA-300, and others - note also NetApp disk trays such as DS4243 and other Xyratex OEM users fit as well)

Note that the power button is just to the inside-front-left (just around the corner from the front-facing LED status lights)

• OEM Kontron Server Motherboard (similar to KTC5520 but without PCI Slot and Sound Card)
• 2 Six-core Xeon E5645 processors @ 2.4ghz
• 96 GB ECC Memory
• Sun MicroSystems ATLS1QGE Quad Port Gigabit Adapter LP Network Card
• Dual 120GB Western Digital Green SSDs (Software RAID-1)
• Avago LSI SAS2008 SAS PCIe JBOD Controller with the following ZFS disk configuration:
  • 12-drive (1TB HGST HUA721010KLA330) single-pool RAIDZ2 (10TB usable) mounted as /peter

Build Notes

1. These are the notes for the build of Landin (and its functional twin Blanton)
2. HW config and notes here: https://wiki.london.hackspace.org.uk/view/Equipment/Landin

Do the right thing and install the Software RAID-1 on the two boot SSDs. Install Notes here SSD install note: NO SWAP PARTITION (we've got 96GB of memory and the SSDs are only 120GB - make a swapfile if we really need on the ZFS array)

Note with the above, grub-install fails, so:

1. fdisk /dev/sda (and then sdb)
2. Add in a second partition that is at the front of the drives, change new partition 2 to type 4 (BIOS BOOT)
3. Then chroot /target /bin/bash and grub-install /dev/sda and grub-install /dev/sdb (assuming these are the SSDs being mirrored)
4. Now system works with grub installs, reboots, etc.

FYI - sda (and similarly sdb) will look like this:

 Disk /dev/sda: 111.8 GiB, 120040980480 bytes, 234455040 sectors
 Units: sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disklabel type: gpt
 Disk identifier: 
 
 Device     Start       End   Sectors   Size Type
 /dev/sda1   2048 234452991 234450944 111.8G Linux RAID
 /dev/sda2     34      2047      2014  1007K BIOS boot

Debian packages to install (support for either legacy multi-user commands, compilation stuff, and more):

Please note you should add "contrib non-free" after main to the /etc/apt/sources.list for ZFS!

iotop htop sudo finger bsdgames ethtool* lynx elinks net-tools openssh-server sudo screen iproute resolvconf build-essential tcpdump vlan ethtool rsync git rdist bzip2 git-core less unzip curl flex bc bison netcat nmap locate vim zsh vim-scripts zfs-dkms zfsutils-linux nfs-kernel-server samba-common-bin qemu-kvm libvirt-clients libvirt-daemon-system libvirt-daemon lshw ipmitool tftpd-hpa apt-mirror smartmontools iozone3 minicom tmux mosh

Show off dmesg

Why can superusers only look at dmesg nowadays? It's kinda useful to see (yeah, OK, fine, everything is a security risk) sudo sysctl kernel.dmesg_restrict=0 kernel.dmesg_restrict = 0 NOTE ABOVE - PUT IN /etc/sysctl.conf to make it permanent.

Installing ZFS, Setting up ZPOOL and Live Disk Swapping

Already setup above in the mega-apt-get command. (Legacy note) Please note you may need to add contrib (and possibly non-free) to the /etc/apt/sources.list (!)

 apt-get install linux-headers-$(uname -r)
 apt-get install zfs-dkms zfsutils-linux

1. EASY WAY TO MAKE THE ZPOOL (NOTE WHETHER YOU WANT RAIDZ1/Z2/Z3 and the WORKING DIRECTORY)
2. Note you're using -f because you're using the whole disk and ignoring legacy disklabels...

 cd /dev/disk/by-id
 sudo zpool create -f peter raidz2  `ls ata-HITACHI*|grep -v part`

(this is easy because all of the donated 1TB drives are same-model HITACHI)

1. FYI - a Similar pool creation expanded out would look like this

 sudo zpool create -f kinnaman raidz2 /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAG06BGA /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAG06EWA /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAG0DJ9A /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ93TMF /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9ES2F /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9GPHF /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9J1EF /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9J59F /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9N1AF /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9N2TF /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PAJ9N3EF /dev/disk/by-id/ata-HITACHI_HUA721010KLA330_PBJ76D4F 

ZFS Disk Death - what to do

If a 1 or 2 disks die in the ZFS zpool, you'll want to replace them. You'll see something like a disk or two with the status UNAVAIL and the zpool state being DEGRADED. We don't want to shut off the computer, so what to do?

• Make note of the disk ID(s) and search for those drives by doing "sudo sas2ircu 0 display | less" and skip to the affected serial number (starts with the letter P in our Hitachi examples)
• Make a note of the enclosure number and the slot number on the controller.
• Make the affected disk(s) blinkk in their slots if you have enclosures that blink properly, such as

 sas2ircu 0 locate 2:1 on #REPLACE WITH THE RIGHT SLOT AND ENCLOSURE ID, DON'T JUST REPLACE THE DRIVES IN EXAMPLE SLOTS
 sas2ircu 0 locate 2:2 on 

then you'll see the blinking slot(s) and can remove those affected disks, replace the drives in the disk trays, and then reinsert.

 sas2ircu 0 locate 2:1 off
 sas2ircu 0 locate 2:2 off

1. Test swapping one disk that's missing/dead for another disk:
2. (you can find out the serial number of the drive that you just swapped by looking in dmesg and then poking around
3. in /dev/disk/by-id/ to find the new drive

Example disk replacement (remember, use zpool status to find the old disk to replace)

 sudo zpool replace -f peter ata-HITACHI_HUA721010KLA330_PAJ9N3EF ata-HITACHI_HUA721010KLA330_PBJ7DNWE

Proxmox setup

1. We'll probably just edit LDAP users to be in that group rather than complicate things with local-remote overlays!
2. libvirt:x:113: and libvirt-qemu:x:64055:
3. Remember to add LDAP users to libvirt group using inheritance connectivity (or we just make the LDAP group be the 'auth'd group'

1. Installed apt-mirror and sync'd the archive from RML's server

- rsync'd various items from RichmondMakerLabs mirror seed, updated /etc/apt/mirror.list with same URLs and updated local disk hierarchy.

TODO

 - PHYSICAL: Move card to proper guaranteed x8 slots and confirm they are negotiating at full 5GT/s (SAS2008 and Sun Quad GBE)
 - crontab zpool scrub (weekly)
 - enable mail sending for daemon support
 x install latest sas2ircu (https://www.broadcom.com/products/storage/host-bus-adapters/sas-9210-8i#downloads) for mgmt
 - install sas2ircu-status (from somewhere else)
 - install bay-identifier.sh
 - label drive trays with last 4 or 6 serial number chunks
 - Play with bayidentifier.sh to see whether we can easily identify the disks in the right bays
 - Play with sas2ircu to see if we can get drives in certain bays to flash (useful finding failed drives to replace)
 - configure smartd and other warning devices
 - integrate into Hackspace infra (automatic emails, root ssh keys, etc.)
 - Find rails to mount into
 - Configure LACP for 4xGbE
 - Export NFS to certain systems over LACP link?i
 - Configure ZeD - /etc/zfs/zed.d/zed.rc Good notes here: http://louwrentius.com/the-zfs-event-daemon-on-linux.html
 - Enable tftpd-hpa for TFTP booting of phones and PXE systems, etc.
 x Enable apt mirroring for Local Debian/Ubuntu installs
 - Documentation for generating VMs
 - Mirroring latest Debian OS for VM installs
 x Add MOTD:

 Welcome to LANDIN.london.hackpsace.org.uk (Debian 9)
 
            NmmdhhhhhhhhhhdmmN          This system provides: 
         mmhhhhhhhhhhhhhhhhhhhhmm       VM hosting for infra & test 
      NmdhhhhhhhhhhhhhhhhhhhhhhhhdmN     - ACNODE -ADMINSTUFF -BRUCE -CHOMSKY 
     mhhhhhhhhhhhhh/``/hhhhhhhhhhhhhm   NFS / TFTP / SMB / OpenLDAP
   Ndhhhhhhhhhhhh/`    `/hhhhhhhhhhhhdN ZFS Volumes & Replication 
  Nhhhhhhhhhhhh/`        ohhhhhhhhhhhhhN 
 Ndhhhhhhhhhhss.          .shhhhhhhhhhhdN Please use CHOMSKY for your general
 dhhhhhhhhh/` .os. ``       .syhhhhhhhhhd system needs. 
 hhhhhhhh/`     .ssy/          `/hhhhhhhh
 hhhhhh/`       .s/              `/hhhhhh
 hhhhhh/`              -o.       `/hhhhhh To create a new VM:
 hhhhhhhh/`          -oss.     `/hhhhhhhh 1. Make Dabley Dobbles.
 dhhhhhhhhhys.       `` .os. `/hhhhhhhhhd 2. Bootle lambda frogs
 Ndhhhhhhhhhhhs.          .sshhhhhhhhhhdN 3. Baz barrybondaboo
  Nhhhhhhhhhhhhho        `/hhhhhhhhhhhhN  4. Edit the wiki with info
   Ndhhhhhhhhhhhh/`    `/hhhhhhhhhhhhdN  
    mhhhhhhhhhhhhh/``/hhhhhhhhhhhhhm    
    NmdhhhhhhhhhhhhhhhhhhhhhhhhdmN     
       mmhhhhhhhhhhhhhhhhhhhhmm        
          NmmdhhhhhhhhhhdmmN           

Storage Pools

As above, one single RAIDZ2 pool of old 1TB 7200 RPM drives known as zpool 'peter'

How to:

Create a new VM (NEED TO REDO WITH NEW PROCEDURE!)

1. Have a login to Landin via LDAP
2. Connect to Landin with your login. (You'll probably need to set up your ssh key first).
3. Create a new VM on Landin. Use local to store the virtual drives
4. Set suitable resources
5. Set network to join bridge br0
6. Start and have fun
7. Add it to this wiki page

Check RAID array status =

(As root):

 doodles doodles.