dataDyne2 - 40T Hypervisor

It’s been 5 years since I created my first storage server and was now approaching full capacity - I was at 90% used out of 9.5TB available. I built a new server with over 40TB usable to replace this server, and will be using the old server for offsite backups eventually.

The new server will run SmartOS like the server it is replacing, and will run almost all of the same zones as the original

Parts

Server parts

20x Hitachi Global Storage 0B26885 UltraStar 4TB SAS2 3.5in 7200RPM 64MB Bare
5x Cable Matters® Mini-SAS to 4 SATA Forward Breakout Cable - 1m
3x SAS9211-8I 8PORT Int 6GB Sata+sas Pcie 2.0
2x Crucial 16GB Kit (8GBx2) DDR3/DDR3L-1600MT/s (PC3-12800) DR x8 ECC UDIMM Server Memory CT2KIT102472BD160B/CT2CP102472BD160B
1x Thermaltake Cooling Fan for Intel Core i7/i5/i3 CL-P0556
1x SUPERMICRO MBD-X9SCL-F-O LGA 1155 Intel C202 Micro ATX Intel Xeon E3 Server Motherboard
1x Intel Xeon Quad-Core Processor E3-1230 v2 3.3GHz 8MB LGA 1155 CPU LGA BX80637E31230V2
1x Corsair CX Series 750 Watt ATX/EPS Modular 80 PLUS Bronze ATX12V/EPS12V 744 Power Supply CX750M

Rack and case parts

Specs

The new build has 20x 4TB SAS drives, 32GB of ECC ram, and an Intel Xeon Quad Core 3.3GHz processor.

Memory

# prtconf | grep Memory
Memory size: 32740 Megabytes

Processor

# psrinfo -vp
The physical processor has 4 cores and 8 virtual processors (0-7)
  The core has 2 virtual processors (0 4)
  The core has 2 virtual processors (1 5)
  The core has 2 virtual processors (2 6)
  The core has 2 virtual processors (3 7)
    x86 (GenuineIntel 306A9 family 6 model 58 step 9 clock 3300 MHz)
      Intel(r) Xeon(r) CPU E3-1230 V2 @ 3.30GHz

Disks (ZFS)

zones 2 disks mirrored for running SmartOS vms (zones)
goliath 18 disks - 2x 9 disk raidz2 for general storage

Status

# zpool status
  pool: goliath
 state: ONLINE
  scan: scrub repaired 0 in 1h57m with 0 errors on Fri Nov 13 05:49:58 2015
config:

        NAME                        STATE     READ WRITE CKSUM
        goliath                     ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            c6t5000CCA03BB55F69d0   ONLINE       0     0     0
            c15t5000CCA03B5E866Dd0  ONLINE       0     0     0
            c7t5000CCA03BAF99BDd0   ONLINE       0     0     0
            c11t5000CCA03BAF8E19d0  ONLINE       0     0     0
            c9t5000CCA03BAEBABDd0   ONLINE       0     0     0
            c0t5000CCA03BB56651d0   ONLINE       0     0     0
            c13t5000CCA03BB47F89d0  ONLINE       0     0     0
            c14t5000CCA03BB46E69d0  ONLINE       0     0     0
            c16t5000CCA03B5C3981d0  ONLINE       0     0     0
          raidz2-1                  ONLINE       0     0     0
            c19t5000CCA03BB4C385d0  ONLINE       0     0     0
            c12t5000CCA03B5E8615d0  ONLINE       0     0     0
            c5t5000CCA03BB04E0Dd0   ONLINE       0     0     0
            c8t5000CCA03B5CF761d0   ONLINE       0     0     0
            c1t5000CCA03BB4C35Dd0   ONLINE       0     0     0
            c4t5000CCA03B5E30DDd0   ONLINE       0     0     0
            c10t5000CCA03BB2B6BDd0  ONLINE       0     0     0
            c17t5000CCA03B5ED5FDd0  ONLINE       0     0     0
            c18t5000CCA03BB47A55d0  ONLINE       0     0     0
        logs
          c21t3d0                   ONLINE       0     0     0

errors: No known data errors

  pool: zones
 state: ONLINE
  scan: scrub repaired 0 in 0h7m with 0 errors on Fri Nov 13 01:08:10 2015
config:

        NAME                       STATE     READ WRITE CKSUM
        zones                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            c3t5000CCA03B5DB449d0  ONLINE       0     0     0
            c2t5000CCA03B5E2C95d0  ONLINE       0     0     0
        logs
          c21t2d0                  ONLINE       0     0     0

errors: No known data errors

Storage Space

# zpool list
NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
goliath    65T  9.23T  55.8T         -     4%    14%  1.00x  ONLINE  -
zones    3.62T  35.8G  3.59T         -     0%     0%  1.00x  ONLINE  -
# zfs list goliath
NAME      USED  AVAIL  REFER  MOUNTPOINT
goliath  7.15T  41.7T  49.6G  /goliath

VMs

# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
1c1855c1-74ae-48c7-ad4c-116fa47ea26c  OS    512      running           ns2
55b6872f-da98-43dc-8545-3674059945d0  OS    512      running           cifs
b334d34d-c6c4-4511-8e0f-16775da0956b  OS    512      running           arbiter
fb9adf5f-d2f8-466f-b4d1-a360f42cf100  OS    512      running           skyevm
02b592d3-ebc6-47a3-806c-5d900e12e79a  OS    1024     running           jabber
0b001125-caf7-44d2-9783-59e207b88711  OS    1024     running           nagios
410b34a1-6441-4d27-85aa-9087d618a16e  LX    1024     running           btsync
75336c33-b4c5-4b7f-9ca5-d109cec4778a  OS    1024     running           vault
ff23f7a5-b0a7-4fb0-aae2-ff57c0afda38  OS    1024     running           ops
7eed36b0-eaee-4c53-98e3-5d1ca97de440  OS    2048     running           vulture
e4c87884-7316-4a47-b617-0f57a2a778b4  LX    2048     running           plex

Pictures

Server open during the installation

The rack and the server

The Reboot stickers are from one of my favorite albums of 2015 - Reboot by The Wonder Girls