Shared storage is a core component of any vmware test lab and there are plenty of free solutions to choose from, FreeNAS and OpenFiler most notably.  But I wanted something that would work on really basic hardware for my backup solution (the scripts for which I’ll be posting here very soon), and I was put off by the many problem posts about the two main solutions when used with ESX.

With a 1TB drive on the shelf already, my goal was to build a shared storage solution for the test lab literally for nothing, primarily for making backups, but also a solution that would scale to be capable, with some just end-of-life enterprise kit, of providing shared storage for running VMs in a DR site.  Could it be done?

NFS Datastores

The market for shared-storage with vmware favours iSCSI, but NFS is just as efficient (more so in some ways), is proven technology, and is available with almost all Linux distributions.  It also doesn’t suffer from the annoying 2TB VMFS limit.  Bluearc have an informative ‘myths dispelled’ sheet for NFS here.

NFS server performance scales well when carefully configured – something reasonably modern and dual-core should easily saturate gigabit with sequential workloads when running on RAID.

A Simple NFS Server

I’ve tried many distributions for this project, but I’ve settled on Debian as an NFS server for ESX because it’s light and because it’s easy to find good ‘how-to’ guides for it.  Perhaps most importantly, so far it’s proved rock-solid as a datastore for ESX both in this tiny backup build, but also on enterprise kit serving up real VMs doing real work, from simple file-and-print to Oracle 11.

A Completely Free NFS Server

700MHz P-III provides about 10MB/s NFS throughput

For my backup build, the goal was basically: free.  A rummage in the loft produced an old Pentium-III HTPC, which looked ideal being small and quiet.

Obviously reliability is a concern.  For better spec hardware that can run ESXi (as in my DR site scenario), using ESXi as the base OS provides excellent health (and performance) monitoring out-the-box, which can be periodically checked using a reporting script such as esx-health.pl.

But for this build, I wanted to keep an eye on the ageing PSU and of course the disk SMART status.  This is handled with lm-sensors, sensord, smartmontools and logcheck packages and email alerting using ssmtp MTA.  Given how invisible these metrics remain with anything from Redmond, the detail available on a ten-year-old standard desktop board really surprised me.  Detailed step-by-step in the wiki.

The Debian Build

The Debian build for NFS is pretty straightforward in the end.  There are a some good gains to be had from tuning options, but mostly it works as-is.

One problem was with the timeout values on ESXi.  The Pentium III and single SATA drive work OK but creating and deleting the super-large VMDK files – particularly a 460GB VMDK that the box has been built to back up – just takes too long and ESXi aborts the tasks with an error.

After a lot of messing about tuning different aspects of NFS, I was able to resolve the file create timeout by switching to the JFS file system, which is a lot less demanding on the hardware.  File delete remains a problem for very large VMDKs for my ancient machine, but I’ve side-stepped that in my backup routines by using plink to delete old files directly over SSH.

The Numbers

Measured using my usual methods, this ‘free’ build has sequential read or write performance of 11MB/s and 8K random of 140 IOPS.  The system has proved stable, completing the 460GB backup in 13 hours without any fuss with an average throughput of 10MB/s.  Power consumption is 30w idle and 44W at full load, and Wake On LAN works so it only runs while a backup is actually being performed (once a week).

For the DR site with PowerEdge 2950 servers I’ve built three configurations, all easily able to saturate gigabit Ethernet on sequential workloads.  All have 8GB RAM, quad-core CPUs and Perc-6i RAID controllers:

• 4x WD 1TB RE3 drives (RAID-10) providing about 1.7TB usable.  Idle at about 180W, 200W full load.  700 8K random IOPS.
• 6x WD 2TB RE4 drives as 3x mirrors.  Three ESXi datastores, three VMDKs presented to the Debian VM for data volumes, and Debian configured to run RAID-0 across these, effectively providing RAID-10.  RAID-10 can’t be used on the Perc controller, because it cannot divide RAID-10 into LUNs of 2TB or less.
• 6x SAS 15k’s (RAID-10) providing about 1TB usable.  Idle about 200W, 220W full load.  Around 1,500 8K random IOPS

In Summary

Debian can be booted from a standard 2GB SD Card

The same Debian Linux build can be used to provide NFS storage to ESXi for everything from a tiny single-drive NFS server to production/DR systems.  Being plain Linux, everything is free, including most of the hardware in my case – the total cost of this build was just £26, and £20 of that was because I decided to boot it from an SD card.

The super-large files used by ESX represent something of a problem to low spec hardware as ESX will timeout file create and file delete operations quite quickly, but JFS helps significantly on low-spec hardware.  More reasonable hardware doesn’t have this issue.