Archive:Equipment/Borg
(→Naming) |
(→Access) |
||
| Line 33: | Line 33: | ||
= Access = | = Access = | ||
Access is through ssh. This system requires physically plugging in the power cable and launching the complex through the RAS controller on Borg5 | |||
= Specs, Layout and Status = | = Specs, Layout and Status = | ||
Revision as of 10:41, 25 June 2015
| Borg | |
|---|---|
 | |
| Model | IBM 3950 M2 |
| Sub-category | Systems |
| Status | Under Construction |
| Accessories | Bits |
| Training requirement | yes |
| ACnode | no |
| Owner | Hackspace |
| Origin | Donation |
| Location | Basement rack |
| Maintainers | Oni |
Borg is a stack of 4 linked machines that combine as one, resulting in half a terabyte of RAM and 64 Cores. It runs Linux and is designed for High Performance Computing on computationally expensive tasks.
Status
The BORG cluster is now complete. The system is running ubuntu 12.04 LTS. We have 4 units in place (the maximum amount) with linux 12.04LTS installed with the usual hackspace credentials.
There are still some tasks to perform before the system is hackspace friendly. See the todo list below.
Naming
We are naming this machine after Anita Borg. Also the Borg.
They're currently labeled as BORG1 through BORG6
Access
Access is through ssh. This system requires physically plugging in the power cable and launching the complex through the RAS controller on Borg5
Specs, Layout and Status
IBM 3950 M2
- 16 cores total
- 4 x Xeon 7330: 4 cores @ 2.4 GHz "Tigerton" Intel Ark spec sheet
- 128 GB ram
The list reflects the layout in the rack.
- BORG5 - 4 CPUs, 128GB RAM - SN 99C5979 - 1.16 BIOS - HEAD Node
- BORG3 - 4 CPUs, 128GB RAM - SN 99C5980 - 1.16 BIOS
- BORG6 - 4 CPUs, 128GB RAM - 1.16 BIOS
- BORG4 - 4 CPUs, 128GB RAM - SN 99B3501 - 1.16 BIOS
BORG1 - 3 CPUs, 8GB RAMBORG2 - 3 CPUs, 8GB RAM
BORGs one and two are lower spec and cannot be linked to the cluster (max of 4 nodes) so should be cannibalised for spares and disposed of.
IP
172.31.24.11
RAS
The RAS II is a separate control system that is setup in the bios and accessible as soon as a BORG unit has power (i.e, if the green light is flashing OR solid). Using a web browser head to either
- Borg3/Borg6 http://172.31.24.170
- Borg5 http://172.31.24.171
Username USERID password PASSW0RD
ScaleXpandr
In order to link upto 4 borgs together we need to use the special cables. http://www.redbooks.ibm.com/redbooks/pdfs/sg247630.pdf - page 235 shows how this is done to create an SMP style set of nodes.
Logbook
Running Debian
- Needs the non-free bnx2 firmware on a flash drive for the install to work, (could try to add it to the install initrd)
- Something something IBM Calgary IOMMU something something leads to DMA errors and the LSI MegaRaid raid card dosn't work, booting with "iommu=soft" makes it work but may not be ideal. search https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt for iommu
- The incantation seems to be: iommu=soft,calgary megaraid_sas.msix_disable=1
Upgrading the bios
Do a diskless boot, the go do Debian and then "Jessie amd64 Diskless for BORGs", log in as root (password 'root', this diskless setup is for testing only!), then:
cd ibm-bios/z/ ./lflash64
This is an upgrade to bios version 1.16
We also need to upgrade:
The RSA II thing: https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086633 The FPGA (in the scaleXpander?) The BMC
We might be able to do it individually, or perhaps use the IBM UpdateXpress thing, which needs SUSE Linux Enterprise Server 11 x86-64 or Red Hat Enterprise Linux 6 x86-64.
We need to reset all the bios etc settings to there defaults.
link dump
- https://bugs.launchpad.net/ubuntu/+source/linux/+bug/343749
- http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5083138
- http://dump.asiantuntijakaveri.fi/le_bueno_dumpo/lsi/
- https://wiki.debian.org/LinuxRaidForAdmins
- http://hwraid.le-vert.net/wiki/DebianPackages
good megaraid cli guide:
- http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
- http://www.linuxsa.org.au/pipermail/linuxsa/2011-November/094772.html
reflashing?!?:
Usage
Energy Consumption
Recent tests comparing Borg3 with AWS , rendering 30 Frames of a small Blender Scene with Blender 2.76 and maxing out the processors:
AWS
- $0.520 per On Demand Linux c1.xlarge Instance Hour at 12 hours
- Cost is roughly £4.60 at current exchange
- Spot instances are possible with Brenda or similar - at $0.07 the cost would be roughly 53p
Borg3
- £0.12p per Kilowatt Hour
- With all 16 Cores operating at 100% Borg was drawing 560Watts
- To render the same scene took 206minutes
- Total Cost 23p
Borg (being a server) could add substantially to our electricity costs. To help this we can offset some of it by the retiring of other machines in the space. It has also been agreed that if it takes up more than 275W regularly then a decision about Borgs future will have to be made. If it goes over 300W we will have to re-think our strategy. These figures are a maximum and equate to about £27 per month in electricity costs. The power consumption will be regularly measured and anyone is welcome to report power consumption levels.
Location
The Rack in the basement.
TODO
- Work out whats up with the 4th missing raid controller
- Play with the raid management thingy
- https://github.com/chicks-net/megamap
- Work out what disks we have and where they are and what they do
- sdc and sdd are now free. (was a vmware install for emf)
- Re-install with debian?
- Get it under ansible (with nfs home dirs and ldap users)
- remove the FC cards we don't need.
- channel bond some of the nics
- do something with the 10Gb Nic's?!?!?
Potential uses
- Rendering video and 3D
- Bio-informatics number crunching (bio-hackers?)
- simulation
- realtime ray-tracing
- Radio FFT decoding in real-time (Cubesat related)
Showing 2 related entities.
