NAS Build (Part II)

(This is a continuation from my previous post ...)

In this post I'll be covering the following,

• Software setup 
• Open Media Vault 
• Software Raid using `mdadm`
• Expectation vs Reality
• Raid setup failure

I received my hardware. Happy days ...

Before narrowing down the hardware I was looking at OMV (OpenMediaVault) and FreeNAS. Newer versions of FreeNAS needs a 64 bit processor and at a minimum 8 gigs of memory. No need to say I went with OMV. 

Software Setup

Open Media Vault

OMV has an images for single board computers (SBC). This made installing OMV in the Pi a peace of cake. There are so many good and even more bad videos on how to do this. I recommend this by TDL.

There are a few gotchas to watch out for. First of all get the correct image for your board. Second of all patience is a virtue. OMV images for SBCs go through a setup in the first boot. If the first boot is unable to complete the setup your image is toast. Make sure you have wired network available to your board and also did I mention patience is a virtue. My initial setup took a little over 10 minutes but word on the street is it can take up to 20 minutes. If you have a monitor connected to the board you will see the control panel address when the setup is complete.

After the setup I did the obvious things like changing the password, setting up ssh, enabling SMB, upgrading all the packages etc. The video I linked does a good job of explaining how to do these. 

Software Raid

OMV has a UI for creating RAID systems but it doesn't allow creating RAIDs out of USB drives (with good reason)

So I opted to use mdadm to create a software raid using my USB drives. I elected to go with a RAID5 setup. As mentioned earlier I have 3 USB HDDs of 1TB size. I can create a 2TB raid disk using RAID5 and be good even if one drive fails. Raid 5 uses a distributed parity system where it stores parity data randomly across all 3 drives. That means if one drive fails the data in that drive can be recovered using the parity information stored in the other two drives. Plus I get to have a 2TB disk 😊. So I ordered another 2TB disk which I'm gonna use as a manual geo redundant backup 😁. 

Here are the steps I used to create the RAID5

First you need to find your disk information. The following command will help you get that

• lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

Then you need to create the raid. The disk information goes into this command as an input

• sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda /dev/sdb /dev/sdc

This is going to take some time. For me it took about 6 hours. the progress of this operation can be viewed by looking at the `/proc/mdstat` file. You need to print the content of the file in order to view the progress. It's overwritten always so a command like `tail` will not work.

• cat /proc/mdstat

Once the raid is created you will see something like this on your `mdstat` file.

Now is the time to tell linux about your RAID array. You do that by adding the raid information to the `mdadm.conf` file. In the below command I'm using `mdadm` to scan the active array and pipe the output to the `mdadm.conf` file using `tee` command

• sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Next we update the initial RAM file system so that the array is available during boot

• sudo update-initramfs -u 

Now if you go back to the OMV RAID management page you should see the newly created RAID array. Next step is to create the partitions and file systems in the raid array. You can do this in the `File Systems` tab in OMV. The whole process is intuitive so I won't go into details.

Now check your `/etc/fstab` file. I found out that OMV added my raid file system information here using labels. I changed it to use name instead. It just comes down to your preference.

There are a lot of info about RAID in the wild. I found this article to be the most helpful.

Expectation vs Reality

At this point I had both the hardware and software set up. I went ahead and created a few SMB shares with custom permissions and went straight into testing the system. I chose the SMB option because I have both Linux and Windows machines in my network. If you need more help in setting up SMB I would recommend this video form TDL.

The first thing I did was to issue a reboot command. The Pi hanged at boot. After some reading I found out that the Pi 4 has a problem with powered USB hubs. It fails to boot when the hub back powers the Pi. This is a known issue. People have had mixed experiences by disconnecting the back powering from the USB hub. I chose not to go down this path due to a few reasons. 

In order to monkey around with the hubs circuit board I need my electronics kit, which I left back at home in Melbourne when I moved to Seattle 🙉. I'm not planning to reboot this setup often so I chose to momentarily unplug the hubs every time I reboot.

Then comes the biggest shock. Before setting up all this I checked the speeds. I was getting between 8/12Mbps using a simple SMB setup between the PI and my laptop. So I expected to get the same speeds with the RAID5. Theoretically a RAID5 should give higher write speeds. In my case I knew the HDD speed was not the bottleneck for 12Mbps  I was getting. The HDDs I used is capable of operating at around 32Mbps. So I wasn't expecting the 12Mbps to go up with the RAID5 but I was not ready for the result I saw.

I was only getting about 0.5Mbps. I could not come up with a definitive explanation but it seems the Pi shares the same controller and bus between the USB and Ethernet. I confirmed this by transferring data between my raid and a USB stick directly plugged into the Pi. I was feeling defeated at this point. This is just the tip of the iceberg.

So after realizing my mistake I just let the system run for a few days. Truth be told I was looking for a good way to break the news to Vindy (my wife). She is a tecky herself and a witty one. So I knew I'm gonna hear about this 🥴. Did I say this was just the tip of the iceberg.

I was checking on the system from time to time and I realized the RAID array kept degrading randomly.

One of the 3 HDDs were getting detached frequently. I tried experimenting with various spin down settings to no avail. The first few times I regenerated the RAID by reattaching the missing disk.

• sudo mdadm /dev/md0 -a /dev/sdd 

After a few attempts I knew that this is not going to work. Stay tuned to find out more about my solution...